openssh unix dev - Jun 2020 - Deprecation of scp protocol and improving sftp client

Hello all,

I believe we all can agree that scp is ugly protocol carried for ages
only for its simplicity of its usage and really no dependencies as it
is installed together with every ssh client. But as we have seen
recently, its simplicity and flexibility comes with security issues
[1], it does not have great performance and there is really no
development in there.

Over the years, we still keep recommending people to use sftp instead,
but its api is not that flexible and simple to be usable as a drop-in
replacement in scripts nor for the occasional ad-hoc transfers of few
files from one server to another.

Before I start hacking, I would like to hear some opinions from others,
whether this is something planned, welcomed or whether there are some
good reasons to keep scp alive.

I have in my mind three things/steps that would make it possible:

 * Update sftp client to be drop-in replacement for scp
   (and/or)
 * Change scp to use sftp internally

 * Modify sshd to use some compatibility "scpd" to support old clients

and some time later

 * Remove scp or replace it with a symlink


[1] http://www.openssh.com/txt/release-8.0

Any ideas/comments/suggestions?


Best regards,
-- 
Jakub Jelen
Senior Software Engineer
Security Technologies
Red Hat, Inc.

On Tue, 2020-06-16 at 18:41 +0200, Jakub Jelen wrote:
>  * Change scp to use sftp internally
Sounds like the best to me... given that scp is used in countless of
scripts, etc..

> Any ideas/comments/suggestions?
Probably unrelated, but what I'd be waiting for years is: XATTRs/ACLs
support in SFTP (thus one could eventually get support for them in
things like sshfs)


Cheer,
Chris.

On 2020/06/16 18:41, Jakub Jelen wrote:
> Hello all,
> 
> I believe we all can agree that scp is ugly protocol carried for ages
> only for its simplicity of its usage and really no dependencies as it
> is installed together with every ssh client. But as we have seen
> recently, its simplicity and flexibility comes with security issues
> [1], it does not have great performance and there is really no
> development in there.
> 
> Over the years, we still keep recommending people to use sftp instead,
> but its api is not that flexible and simple to be usable as a drop-in
> replacement in scripts nor for the occasional ad-hoc transfers of few
> files from one server to another.
I've tried to switch to sftp several times, the thing that always
stops me is not being able to copy from local->remote directly
on the command line.
> Before I start hacking, I would like to hear some opinions from others,
> whether this is something planned, welcomed or whether there are some
> good reasons to keep scp alive.
> 
> I have in my mind three things/steps that would make it possible:
> 
>  * Update sftp client to be drop-in replacement for scp
>    (and/or)
This would seem a good starting point. It would allow using
"alias scp=sftp" for interactive shells (muscle memory / easier
typing) and makes it possible to convert scripts across one by one
from scp to sftp without unexpectedly breaking anything.
>  * Change scp to use sftp internally
Then you either have no fallback for scp with very old servers, or a
mess of either/or code to cope with both protocols. And it would seem
more complex to change the protocol code than the UI code.
> 
>  * Modify sshd to use some compatibility "scpd" to support old
clients
> 
> and some time later
> 
>  * Remove scp or replace it with a symlink
> 
> 
> [1] http://www.openssh.com/txt/release-8.0
> 
> Any ideas/comments/suggestions?
> 
> 
> Best regards,
> -- 
> Jakub Jelen
> Senior Software Engineer
> Security Technologies
> Red Hat, Inc.
> 
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

scp may be an ugly protocol, but it works, works nicely from a command line, and
is quite convenient. FTP (and, presumably, sftp) is not nearly as convenient.

Why do you think your recommendation to "use sftp instead" keeps
falling on the deaf ear? Usability, perhaps?

Perhaps it's time to stop preaching to people about what they should use,
but instead - if you really want a change - design, implement, and release a new
tool that would combine the usability of scp with the security you want to see?
SFTP did not seem to make that cut. Next option?



?On 6/16/20, 13:14, "openssh-unix-dev on behalf of Stuart Henderson"
<openssh-unix-dev-bounces+uri=ll.mit.edu at mindrot.org on behalf of stu at
spacehopper.org> wrote:

    On 2020/06/16 18:41, Jakub Jelen wrote:
    > Hello all,
    > 
    > I believe we all can agree that scp is ugly protocol carried for ages
    > only for its simplicity of its usage and really no dependencies as it
    > is installed together with every ssh client. But as we have seen
    > recently, its simplicity and flexibility comes with security issues
    > [1], it does not have great performance and there is really no
    > development in there.
    > 
    > Over the years, we still keep recommending people to use sftp instead,
    > but its api is not that flexible and simple to be usable as a drop-in
    > replacement in scripts nor for the occasional ad-hoc transfers of few
    > files from one server to another.

    I've tried to switch to sftp several times, the thing that always
    stops me is not being able to copy from local->remote directly
    on the command line.

    > Before I start hacking, I would like to hear some opinions from others,
    > whether this is something planned, welcomed or whether there are some
    > good reasons to keep scp alive.
    > 
    > I have in my mind three things/steps that would make it possible:
    > 
    >  * Update sftp client to be drop-in replacement for scp
    >    (and/or)

    This would seem a good starting point. It would allow using
    "alias scp=sftp" for interactive shells (muscle memory / easier
    typing) and makes it possible to convert scripts across one by one
    from scp to sftp without unexpectedly breaking anything.

    >  * Change scp to use sftp internally

    Then you either have no fallback for scp with very old servers, or a
    mess of either/or code to cope with both protocols. And it would seem
    more complex to change the protocol code than the UI code.

    > 
    >  * Modify sshd to use some compatibility "scpd" to support
old clients
    > 
    > and some time later
    > 
    >  * Remove scp or replace it with a symlink
    > 
    > 
    > [1] http://www.openssh.com/txt/release-8.0
    > 
    > Any ideas/comments/suggestions?
    > 
    > 
    > Best regards,
    > -- 
    > Jakub Jelen
    > Senior Software Engineer
    > Security Technologies
    > Red Hat, Inc.
    > 
    > _______________________________________________
    > openssh-unix-dev mailing list
    > openssh-unix-dev at mindrot.org
    > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
    _______________________________________________
    openssh-unix-dev mailing list
    openssh-unix-dev at mindrot.org
    https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5249 bytes
Desc: not available
URL:
<http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20200616/aeec019c/attachment-0001.p7s>

Jakub Jelen wrote:
> Any ideas/comments/suggestions?
I doubt that upstream would drop the scp utility and protocol easily.
There is no point in breaking compatibility, so the scp utility would
stay at the very least for the benefit of remote scp clients.

That said, I think it would be worthwhile to implement or at the very
least research the possibility of an SFTP client with scp semantics.

You'd have to look closely at the protocols and scp man page, and it
could well turn out that the SFTP protocol doesn't currently offer
everything needed to provide full scp compatibility.

I'd be interested to hear what you find, if you go ahead.


//Peter

Agreed with what Uri said on convenience. Being able to use "scp" with
the
same syntax as "cp" is wired into a lot of peoples hands. sftp
doesn't
allow for the same easy syntax. Whatever is done to replace it should allow
for that same easy syntax if you want to see adoption with a minimum of
kicking & screaming.

On Tue, Jun 16, 2020 at 10:38 AM Peter Stuge <peter at stuge.se> wrote:
> Jakub Jelen wrote:
> > Any ideas/comments/suggestions?
>
> I doubt that upstream would drop the scp utility and protocol easily.
> There is no point in breaking compatibility, so the scp utility would
> stay at the very least for the benefit of remote scp clients.
>
> That said, I think it would be worthwhile to implement or at the very
> least research the possibility of an SFTP client with scp semantics.
>
> You'd have to look closely at the protocols and scp man page, and it
> could well turn out that the SFTP protocol doesn't currently offer
> everything needed to provide full scp compatibility.
>
> I'd be interested to hear what you find, if you go ahead.
>
>
> //Peter
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

On 2020/06/16 17:37, Peter Stuge wrote:
> Jakub Jelen wrote:
> > Any ideas/comments/suggestions?
> 
> I doubt that upstream would drop the scp utility and protocol easily.
> There is no point in breaking compatibility
OpenSSH does break compatibility where there's good reason to.
SSH 1, <1024 bit keys.

On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen
wrote:
>  * Change scp to use sftp internally
As an existence proof, pscp (from PuTTY) does exactly this; it tries the
sftp protocol and falls back to the scp protocol if that fails, and it
has -scp or -sftp options to force one or the other.  I've long meant to
get round to putting something similar for OpenSSH, but never got far
enough to really be worth mentioning.  (Of course it would still need to
retain scp "source" and "sink" modes if invoked with -f or
-t, to retain
compatibility, since those are used on the server when an scp client
connects; but there's no particular obstacle to that.)
>  * Modify sshd to use some compatibility "scpd" to support old
clients
This should be unnecessary.  When an scp client connects to an scp
server, it passes the -f (source) or -t (sink) flag as appropriate;
neither is part of the documented user-facing interface to scp.  I don't
see any particular reason why scp (the program) couldn't continue to
speak the scp protocol when invoked with -f/-t, but speak the sftp
protocol when invoked in the normal way.

-- 
Colin Watson (he/him)                              [cjwatson at debian.org]

On Tue, 2020-06-16 at 19:47 +0100, Colin Watson wrote:
> sftp protocol and falls back to the scp protocol if that fails, and
> it
> has -scp or -sftp options to force one or the other.
I think fallback should be something that doesn't happen automatically,
at least at some point.

I mean the whole reason of getting rid of scp is mostly the security
issues, isn't it?


I wonder if an "new" scp could bring in some new features like --
interactive (not overwrite files without askin)... or if that would
break too much

Cheers,
Chris.

On Tuesday, 16 June 2020 20:47:24 CEST Colin Watson
wrote:
> On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen wrote:
> >  * Change scp to use sftp internally
> 
> As an existence proof, pscp (from PuTTY) does exactly this; it tries the
> sftp protocol and falls back to the scp protocol if that fails, and it
> has -scp or -sftp options to force one or the other.  I've long meant
to
> get round to putting something similar for OpenSSH, but never got far
> enough to really be worth mentioning.  (Of course it would still need to
> retain scp "source" and "sink" modes if invoked with -f
or -t, to retain
> compatibility, since those are used on the server when an scp client
> connects; but there's no particular obstacle to that.)
> 
> >  * Modify sshd to use some compatibility "scpd" to support
old clients
> 
> This should be unnecessary.  When an scp client connects to an scp
> server, it passes the -f (source) or -t (sink) flag as appropriate;
> neither is part of the documented user-facing interface to scp.  I
don't
> see any particular reason why scp (the program) couldn't continue to
> speak the scp protocol when invoked with -f/-t, but speak the sftp
> protocol when invoked in the normal way.
It is necessary. If you replace the protocol of the scp command with sftp then 
there are still scp commands out there speaking the scp protocol. You need to 
keep the server side supported.


	Andreas

-- 
Andreas Schneider                 asn at cryptomilk.org
GPG-ID:     8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D

On Tuesday, 16 June 2020 18:41:09 CEST Jakub Jelen
wrote:
> Hello all,
Hi Jakub,
> I have in my mind three things/steps that would make it possible:
> 
>  * Update sftp client to be drop-in replacement for scp
>    (and/or)
>  * Change scp to use sftp internally
I would suggest to go this route and let the `scp` tool use the sftp protocol 
by default. Keep the `sftp` command as is.
>  * Modify sshd to use some compatibility "scpd" to support old
clients
Yes, and add an sshd option to disable it. This way we can flip the option to 
to disabled by default at one point.


	Andreas


-- 
Andreas Schneider                 asn at cryptomilk.org
GPG-ID:     8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D

On Tuesday, 16 June 2020 20:47:24 CEST Colin Watson
wrote:
> On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen wrote:
> >  * Change scp to use sftp internally
> 
> As an existence proof, pscp (from PuTTY) does exactly this; it tries the
> sftp protocol and falls back to the scp protocol if that fails, and it
> has -scp or -sftp options to force one or the other.  I've long meant
to
> get round to putting something similar for OpenSSH, but never got far
> enough to really be worth mentioning.  (Of course it would still need to
> retain scp "source" and "sink" modes if invoked with -f
or -t, to retain
> compatibility, since those are used on the server when an scp client
> connects; but there's no particular obstacle to that.)
You can easily detect the sink mode on connect and redirect to a scpd server 
implementation. There is no need to make it more complex than it is.

We have sftp server implementation around for long enough that the `scp` 
command can use the protocol.

Also if you have a scpd. You can reject the scp protocol completely by a 
config option.
> >  * Modify sshd to use some compatibility "scpd" to support
old clients
> 
> This should be unnecessary.  When an scp client connects to an scp
> server, it passes the -f (source) or -t (sink) flag as appropriate;
> neither is part of the documented user-facing interface to scp.  I
don't
> see any particular reason why scp (the program) couldn't continue to
> speak the scp protocol when invoked with -f/-t, but speak the sftp
> protocol when invoked in the normal way.
The scp command should only handle the client side, for the server you should 
have a server only implementation which could be disabled. Some people are not 
interested in the scp protocol if sftp can do the job. One security hole less 
:-)


	Andreas

-- 
Andreas Schneider                 asn at cryptomilk.org
GPG-ID:     8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D

I had something in mind like this for years, but with slightly different steps:
My naive approach would be to keep the scp user interface and switch
to the sftp protocol internally. We could add a -M [scp|sftp] option
to scp and select the internal protocol. Later we switch the default
from scp to sftp.
No need to change sshd or write scpd.

-m

Am Di., 16. Juni 2020 um 18:48 Uhr schrieb Jakub Jelen <jjelen at
redhat.com>:
>
> Hello all,
>
> I believe we all can agree that scp is ugly protocol carried for ages
> only for its simplicity of its usage and really no dependencies as it
> is installed together with every ssh client. But as we have seen
> recently, its simplicity and flexibility comes with security issues
> [1], it does not have great performance and there is really no
> development in there.
>
> Over the years, we still keep recommending people to use sftp instead,
> but its api is not that flexible and simple to be usable as a drop-in
> replacement in scripts nor for the occasional ad-hoc transfers of few
> files from one server to another.
>
> Before I start hacking, I would like to hear some opinions from others,
> whether this is something planned, welcomed or whether there are some
> good reasons to keep scp alive.
>
> I have in my mind three things/steps that would make it possible:
>
>  * Update sftp client to be drop-in replacement for scp
>    (and/or)
>  * Change scp to use sftp internally
>
>  * Modify sshd to use some compatibility "scpd" to support old
clients
>
> and some time later
>
>  * Remove scp or replace it with a symlink
>
>
> [1] http://www.openssh.com/txt/release-8.0
>
> Any ideas/comments/suggestions?
>
>
> Best regards,
> --
> Jakub Jelen
> Senior Software Engineer
> Security Technologies
> Red Hat, Inc.
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

On Tue, 2020-06-23 at 08:06 +0200, Markus Friedl wrote:
> I had something in mind like this for years, but with slightly
> different steps:
> My naive approach would be to keep the scp user interface and switch
> to the sftp protocol internally. We could add a -M [scp|sftp] option
> to scp and select the internal protocol. Later we switch the default
> from scp to sftp.
> No need to change sshd or write scpd.
Thank you all for comments and initial feedback. It looks like this is
something that many people already had in minds, but for some reasons,
it never happened.

I tried to put together something that now works and passes the scp
testsuite (with both scp and sftp modes):

https://github.com/openssh/openssh-portable/pull/194

What does not work is the extended remote-to-remote through local,
which would require some more low-level protocol tweaks.

Most of the code is taken and adapted from the sftp.c . There are still
a few TODOs, but lets take it as a first iteration/proof of concept.

Considering the automatic fallback to scp, the way how it works now
(executing another ssh in subprocess) it would require going through
the whole key exchange and authentication again, spawning a new ssh
process, which sounds very cumbersome.

Much nicer solution would be using ssh_api to handle commands/subsystem
requests inside a single ssh session, but it would be much more code
and changes that I wanted to avoid in the first iteration (if the
ssh_api is already usable for something like this -- I would have to
check).

So again, comments, suggestions and feedback welcomed. I am not sure if
there is some other mailing list to get more attention from other
OpenBSD developers or this one is fine.

Thanks,
Jakub
> -m
> 
> Am Di., 16. Juni 2020 um 18:48 Uhr schrieb Jakub Jelen <
> jjelen at redhat.com>:
> > Hello all,
> > 
> > I believe we all can agree that scp is ugly protocol carried for
> > ages
> > only for its simplicity of its usage and really no dependencies as
> > it
> > is installed together with every ssh client. But as we have seen
> > recently, its simplicity and flexibility comes with security issues
> > [1], it does not have great performance and there is really no
> > development in there.
> > 
> > Over the years, we still keep recommending people to use sftp
> > instead,
> > but its api is not that flexible and simple to be usable as a drop-
> > in
> > replacement in scripts nor for the occasional ad-hoc transfers of
> > few
> > files from one server to another.
> > 
> > Before I start hacking, I would like to hear some opinions from
> > others,
> > whether this is something planned, welcomed or whether there are
> > some
> > good reasons to keep scp alive.
> > 
> > I have in my mind three things/steps that would make it possible:
> > 
> >  * Update sftp client to be drop-in replacement for scp
> >    (and/or)
> >  * Change scp to use sftp internally
> > 
> >  * Modify sshd to use some compatibility "scpd" to support
old
> > clients
> > 
> > and some time later
> > 
> >  * Remove scp or replace it with a symlink
> > 
> > 
> > [1] http://www.openssh.com/txt/release-8.0
> > 
> > Any ideas/comments/suggestions?
> > 
> > 
> > Best regards,
> > --
> > Jakub Jelen
> > Senior Software Engineer
> > Security Technologies
> > Red Hat, Inc.
> > 
> > _______________________________________________
> > openssh-unix-dev mailing list
> > openssh-unix-dev at mindrot.org
> > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> 
-- 
Jakub Jelen
Senior Software Engineer
Security Technologies
Red Hat, Inc.

The fact that scp has been carried for ages is part of its value
proposition. It?s going to work. Much as I am fond of my dd/cat/tar/smoke
signal hacks, scp is the standard.

A bit of a tangent:  There is a security hole in scp, ssh, TLS, basically
everything running over IP. As transfer sizes have increased significantly,
the identity of a certain population of files being moved is increasingly
obvious due to byte count, compressed or otherwise.

No, you don?t get to pad every transfer out to the maximum possible size,
or delay transfers until you can aggregate payloads. Yes, somebody thought
of that.

There are limits.

On Tue, Jun 16, 2020 at 9:47 AM Jakub Jelen <jjelen at redhat.com> wrote:
> Hello all,
>
> I believe we all can agree that scp is ugly protocol carried for ages
> only for its simplicity of its usage and really no dependencies as it
> is installed together with every ssh client. But as we have seen
> recently, its simplicity and flexibility comes with security issues
> [1], it does not have great performance and there is really no
> development in there.
>
> Over the years, we still keep recommending people to use sftp instead,
> but its api is not that flexible and simple to be usable as a drop-in
> replacement in scripts nor for the occasional ad-hoc transfers of few
> files from one server to another.
>
> Before I start hacking, I would like to hear some opinions from others,
> whether this is something planned, welcomed or whether there are some
> good reasons to keep scp alive.
>
> I have in my mind three things/steps that would make it possible:
>
>  * Update sftp client to be drop-in replacement for scp
>    (and/or)
>  * Change scp to use sftp internally
>
>  * Modify sshd to use some compatibility "scpd" to support old
clients
>
> and some time later
>
>  * Remove scp or replace it with a symlink
>
>
> [1] http://www.openssh.com/txt/release-8.0
>
> Any ideas/comments/suggestions?
>
>
> Best regards,
> --
> Jakub Jelen
> Senior Software Engineer
> Security Technologies
> Red Hat, Inc.
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

The fundamental problem is that the command line syntax of sftp is not 
great. There's no fundamental reason that can't be fixed, without even 
changing the existing sftp protocol. But as long as scp (and rsync) 
exist the motivation to actually make the sftp command line tool nicer 
is so low that even though people have talked about it for 20 years, it 
hasn't happened yet. (And even though it would require orders of 
magnitude less effort than has been expended complaining about sftp & 
scp over the last 20 years...)

I don't see an obvious way forward until someone decides to actually 
make the sftp client nicer to use. (No need to even talk about it: just 
show up with a finished version that does what scp does!)