Hello all, I believe we all can agree that scp is ugly protocol carried for ages only for its simplicity of its usage and really no dependencies as it is installed together with every ssh client. But as we have seen recently, its simplicity and flexibility comes with security issues [1], it does not have great performance and there is really no development in there. Over the years, we still keep recommending people to use sftp instead, but its api is not that flexible and simple to be usable as a drop-in replacement in scripts nor for the occasional ad-hoc transfers of few files from one server to another. Before I start hacking, I would like to hear some opinions from others, whether this is something planned, welcomed or whether there are some good reasons to keep scp alive. I have in my mind three things/steps that would make it possible: * Update sftp client to be drop-in replacement for scp (and/or) * Change scp to use sftp internally * Modify sshd to use some compatibility "scpd" to support old clients and some time later * Remove scp or replace it with a symlink [1] http://www.openssh.com/txt/release-8.0 Any ideas/comments/suggestions? Best regards, -- Jakub Jelen Senior Software Engineer Security Technologies Red Hat, Inc.
Christoph Anton Mitterer
2020-Jun-16 16:54 UTC
Deprecation of scp protocol and improving sftp client
On Tue, 2020-06-16 at 18:41 +0200, Jakub Jelen wrote:> * Change scp to use sftp internallySounds like the best to me... given that scp is used in countless of scripts, etc..> Any ideas/comments/suggestions?Probably unrelated, but what I'd be waiting for years is: XATTRs/ACLs support in SFTP (thus one could eventually get support for them in things like sshfs) Cheer, Chris.
Stuart Henderson
2020-Jun-16 17:10 UTC
Deprecation of scp protocol and improving sftp client
On 2020/06/16 18:41, Jakub Jelen wrote:> Hello all, > > I believe we all can agree that scp is ugly protocol carried for ages > only for its simplicity of its usage and really no dependencies as it > is installed together with every ssh client. But as we have seen > recently, its simplicity and flexibility comes with security issues > [1], it does not have great performance and there is really no > development in there. > > Over the years, we still keep recommending people to use sftp instead, > but its api is not that flexible and simple to be usable as a drop-in > replacement in scripts nor for the occasional ad-hoc transfers of few > files from one server to another.I've tried to switch to sftp several times, the thing that always stops me is not being able to copy from local->remote directly on the command line.> Before I start hacking, I would like to hear some opinions from others, > whether this is something planned, welcomed or whether there are some > good reasons to keep scp alive. > > I have in my mind three things/steps that would make it possible: > > * Update sftp client to be drop-in replacement for scp > (and/or)This would seem a good starting point. It would allow using "alias scp=sftp" for interactive shells (muscle memory / easier typing) and makes it possible to convert scripts across one by one from scp to sftp without unexpectedly breaking anything.> * Change scp to use sftp internallyThen you either have no fallback for scp with very old servers, or a mess of either/or code to cope with both protocols. And it would seem more complex to change the protocol code than the UI code.> > * Modify sshd to use some compatibility "scpd" to support old clients > > and some time later > > * Remove scp or replace it with a symlink > > > [1] http://www.openssh.com/txt/release-8.0 > > Any ideas/comments/suggestions? > > > Best regards, > -- > Jakub Jelen > Senior Software Engineer > Security Technologies > Red Hat, Inc. > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
Blumenthal, Uri - 0553 - MITLL
2020-Jun-16 17:20 UTC
Deprecation of scp protocol and improving sftp client
scp may be an ugly protocol, but it works, works nicely from a command line, and is quite convenient. FTP (and, presumably, sftp) is not nearly as convenient. Why do you think your recommendation to "use sftp instead" keeps falling on the deaf ear? Usability, perhaps? Perhaps it's time to stop preaching to people about what they should use, but instead - if you really want a change - design, implement, and release a new tool that would combine the usability of scp with the security you want to see? SFTP did not seem to make that cut. Next option? ?On 6/16/20, 13:14, "openssh-unix-dev on behalf of Stuart Henderson" <openssh-unix-dev-bounces+uri=ll.mit.edu at mindrot.org on behalf of stu at spacehopper.org> wrote: On 2020/06/16 18:41, Jakub Jelen wrote: > Hello all, > > I believe we all can agree that scp is ugly protocol carried for ages > only for its simplicity of its usage and really no dependencies as it > is installed together with every ssh client. But as we have seen > recently, its simplicity and flexibility comes with security issues > [1], it does not have great performance and there is really no > development in there. > > Over the years, we still keep recommending people to use sftp instead, > but its api is not that flexible and simple to be usable as a drop-in > replacement in scripts nor for the occasional ad-hoc transfers of few > files from one server to another. I've tried to switch to sftp several times, the thing that always stops me is not being able to copy from local->remote directly on the command line. > Before I start hacking, I would like to hear some opinions from others, > whether this is something planned, welcomed or whether there are some > good reasons to keep scp alive. > > I have in my mind three things/steps that would make it possible: > > * Update sftp client to be drop-in replacement for scp > (and/or) This would seem a good starting point. It would allow using "alias scp=sftp" for interactive shells (muscle memory / easier typing) and makes it possible to convert scripts across one by one from scp to sftp without unexpectedly breaking anything. > * Change scp to use sftp internally Then you either have no fallback for scp with very old servers, or a mess of either/or code to cope with both protocols. And it would seem more complex to change the protocol code than the UI code. > > * Modify sshd to use some compatibility "scpd" to support old clients > > and some time later > > * Remove scp or replace it with a symlink > > > [1] http://www.openssh.com/txt/release-8.0 > > Any ideas/comments/suggestions? > > > Best regards, > -- > Jakub Jelen > Senior Software Engineer > Security Technologies > Red Hat, Inc. > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev _______________________________________________ openssh-unix-dev mailing list openssh-unix-dev at mindrot.org https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5249 bytes Desc: not available URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20200616/aeec019c/attachment-0001.p7s>
Jakub Jelen wrote:> Any ideas/comments/suggestions?I doubt that upstream would drop the scp utility and protocol easily. There is no point in breaking compatibility, so the scp utility would stay at the very least for the benefit of remote scp clients. That said, I think it would be worthwhile to implement or at the very least research the possibility of an SFTP client with scp semantics. You'd have to look closely at the protocols and scp man page, and it could well turn out that the SFTP protocol doesn't currently offer everything needed to provide full scp compatibility. I'd be interested to hear what you find, if you go ahead. //Peter
Agreed with what Uri said on convenience. Being able to use "scp" with the same syntax as "cp" is wired into a lot of peoples hands. sftp doesn't allow for the same easy syntax. Whatever is done to replace it should allow for that same easy syntax if you want to see adoption with a minimum of kicking & screaming. On Tue, Jun 16, 2020 at 10:38 AM Peter Stuge <peter at stuge.se> wrote:> Jakub Jelen wrote: > > Any ideas/comments/suggestions? > > I doubt that upstream would drop the scp utility and protocol easily. > There is no point in breaking compatibility, so the scp utility would > stay at the very least for the benefit of remote scp clients. > > That said, I think it would be worthwhile to implement or at the very > least research the possibility of an SFTP client with scp semantics. > > You'd have to look closely at the protocols and scp man page, and it > could well turn out that the SFTP protocol doesn't currently offer > everything needed to provide full scp compatibility. > > I'd be interested to hear what you find, if you go ahead. > > > //Peter > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >
Stuart Henderson
2020-Jun-16 17:51 UTC
Deprecation of scp protocol and improving sftp client
On 2020/06/16 17:37, Peter Stuge wrote:> Jakub Jelen wrote: > > Any ideas/comments/suggestions? > > I doubt that upstream would drop the scp utility and protocol easily. > There is no point in breaking compatibilityOpenSSH does break compatibility where there's good reason to. SSH 1, <1024 bit keys.
On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen wrote:> * Change scp to use sftp internallyAs an existence proof, pscp (from PuTTY) does exactly this; it tries the sftp protocol and falls back to the scp protocol if that fails, and it has -scp or -sftp options to force one or the other. I've long meant to get round to putting something similar for OpenSSH, but never got far enough to really be worth mentioning. (Of course it would still need to retain scp "source" and "sink" modes if invoked with -f or -t, to retain compatibility, since those are used on the server when an scp client connects; but there's no particular obstacle to that.)> * Modify sshd to use some compatibility "scpd" to support old clientsThis should be unnecessary. When an scp client connects to an scp server, it passes the -f (source) or -t (sink) flag as appropriate; neither is part of the documented user-facing interface to scp. I don't see any particular reason why scp (the program) couldn't continue to speak the scp protocol when invoked with -f/-t, but speak the sftp protocol when invoked in the normal way. -- Colin Watson (he/him) [cjwatson at debian.org]
Christoph Anton Mitterer
2020-Jun-16 20:14 UTC
Deprecation of scp protocol and improving sftp client
On Tue, 2020-06-16 at 19:47 +0100, Colin Watson wrote:> sftp protocol and falls back to the scp protocol if that fails, and > it > has -scp or -sftp options to force one or the other.I think fallback should be something that doesn't happen automatically, at least at some point. I mean the whole reason of getting rid of scp is mostly the security issues, isn't it? I wonder if an "new" scp could bring in some new features like -- interactive (not overwrite files without askin)... or if that would break too much Cheers, Chris.
Andreas Schneider
2020-Jun-22 08:47 UTC
Deprecation of scp protocol and improving sftp client
On Tuesday, 16 June 2020 20:47:24 CEST Colin Watson wrote:> On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen wrote: > > * Change scp to use sftp internally > > As an existence proof, pscp (from PuTTY) does exactly this; it tries the > sftp protocol and falls back to the scp protocol if that fails, and it > has -scp or -sftp options to force one or the other. I've long meant to > get round to putting something similar for OpenSSH, but never got far > enough to really be worth mentioning. (Of course it would still need to > retain scp "source" and "sink" modes if invoked with -f or -t, to retain > compatibility, since those are used on the server when an scp client > connects; but there's no particular obstacle to that.) > > > * Modify sshd to use some compatibility "scpd" to support old clients > > This should be unnecessary. When an scp client connects to an scp > server, it passes the -f (source) or -t (sink) flag as appropriate; > neither is part of the documented user-facing interface to scp. I don't > see any particular reason why scp (the program) couldn't continue to > speak the scp protocol when invoked with -f/-t, but speak the sftp > protocol when invoked in the normal way.It is necessary. If you replace the protocol of the scp command with sftp then there are still scp commands out there speaking the scp protocol. You need to keep the server side supported. Andreas -- Andreas Schneider asn at cryptomilk.org GPG-ID: 8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D
Andreas Schneider
2020-Jun-22 08:52 UTC
Deprecation of scp protocol and improving sftp client
On Tuesday, 16 June 2020 18:41:09 CEST Jakub Jelen wrote:> Hello all,Hi Jakub,> I have in my mind three things/steps that would make it possible: > > * Update sftp client to be drop-in replacement for scp > (and/or) > * Change scp to use sftp internallyI would suggest to go this route and let the `scp` tool use the sftp protocol by default. Keep the `sftp` command as is.> * Modify sshd to use some compatibility "scpd" to support old clientsYes, and add an sshd option to disable it. This way we can flip the option to to disabled by default at one point. Andreas -- Andreas Schneider asn at cryptomilk.org GPG-ID: 8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D
Andreas Schneider
2020-Jun-22 08:58 UTC
Deprecation of scp protocol and improving sftp client
On Tuesday, 16 June 2020 20:47:24 CEST Colin Watson wrote:> On Tue, Jun 16, 2020 at 06:41:09PM +0200, Jakub Jelen wrote: > > * Change scp to use sftp internally > > As an existence proof, pscp (from PuTTY) does exactly this; it tries the > sftp protocol and falls back to the scp protocol if that fails, and it > has -scp or -sftp options to force one or the other. I've long meant to > get round to putting something similar for OpenSSH, but never got far > enough to really be worth mentioning. (Of course it would still need to > retain scp "source" and "sink" modes if invoked with -f or -t, to retain > compatibility, since those are used on the server when an scp client > connects; but there's no particular obstacle to that.)You can easily detect the sink mode on connect and redirect to a scpd server implementation. There is no need to make it more complex than it is. We have sftp server implementation around for long enough that the `scp` command can use the protocol. Also if you have a scpd. You can reject the scp protocol completely by a config option.> > * Modify sshd to use some compatibility "scpd" to support old clients > > This should be unnecessary. When an scp client connects to an scp > server, it passes the -f (source) or -t (sink) flag as appropriate; > neither is part of the documented user-facing interface to scp. I don't > see any particular reason why scp (the program) couldn't continue to > speak the scp protocol when invoked with -f/-t, but speak the sftp > protocol when invoked in the normal way.The scp command should only handle the client side, for the server you should have a server only implementation which could be disabled. Some people are not interested in the scp protocol if sftp can do the job. One security hole less :-) Andreas -- Andreas Schneider asn at cryptomilk.org GPG-ID: 8DFF53E18F2ABC8D8F3C92237EE0FC4DCC014E3D
I had something in mind like this for years, but with slightly different steps: My naive approach would be to keep the scp user interface and switch to the sftp protocol internally. We could add a -M [scp|sftp] option to scp and select the internal protocol. Later we switch the default from scp to sftp. No need to change sshd or write scpd. -m Am Di., 16. Juni 2020 um 18:48 Uhr schrieb Jakub Jelen <jjelen at redhat.com>:> > Hello all, > > I believe we all can agree that scp is ugly protocol carried for ages > only for its simplicity of its usage and really no dependencies as it > is installed together with every ssh client. But as we have seen > recently, its simplicity and flexibility comes with security issues > [1], it does not have great performance and there is really no > development in there. > > Over the years, we still keep recommending people to use sftp instead, > but its api is not that flexible and simple to be usable as a drop-in > replacement in scripts nor for the occasional ad-hoc transfers of few > files from one server to another. > > Before I start hacking, I would like to hear some opinions from others, > whether this is something planned, welcomed or whether there are some > good reasons to keep scp alive. > > I have in my mind three things/steps that would make it possible: > > * Update sftp client to be drop-in replacement for scp > (and/or) > * Change scp to use sftp internally > > * Modify sshd to use some compatibility "scpd" to support old clients > > and some time later > > * Remove scp or replace it with a symlink > > > [1] http://www.openssh.com/txt/release-8.0 > > Any ideas/comments/suggestions? > > > Best regards, > -- > Jakub Jelen > Senior Software Engineer > Security Technologies > Red Hat, Inc. > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
On Tue, 2020-06-23 at 08:06 +0200, Markus Friedl wrote:> I had something in mind like this for years, but with slightly > different steps: > My naive approach would be to keep the scp user interface and switch > to the sftp protocol internally. We could add a -M [scp|sftp] option > to scp and select the internal protocol. Later we switch the default > from scp to sftp. > No need to change sshd or write scpd.Thank you all for comments and initial feedback. It looks like this is something that many people already had in minds, but for some reasons, it never happened. I tried to put together something that now works and passes the scp testsuite (with both scp and sftp modes): https://github.com/openssh/openssh-portable/pull/194 What does not work is the extended remote-to-remote through local, which would require some more low-level protocol tweaks. Most of the code is taken and adapted from the sftp.c . There are still a few TODOs, but lets take it as a first iteration/proof of concept. Considering the automatic fallback to scp, the way how it works now (executing another ssh in subprocess) it would require going through the whole key exchange and authentication again, spawning a new ssh process, which sounds very cumbersome. Much nicer solution would be using ssh_api to handle commands/subsystem requests inside a single ssh session, but it would be much more code and changes that I wanted to avoid in the first iteration (if the ssh_api is already usable for something like this -- I would have to check). So again, comments, suggestions and feedback welcomed. I am not sure if there is some other mailing list to get more attention from other OpenBSD developers or this one is fine. Thanks, Jakub> -m > > Am Di., 16. Juni 2020 um 18:48 Uhr schrieb Jakub Jelen < > jjelen at redhat.com>: > > Hello all, > > > > I believe we all can agree that scp is ugly protocol carried for > > ages > > only for its simplicity of its usage and really no dependencies as > > it > > is installed together with every ssh client. But as we have seen > > recently, its simplicity and flexibility comes with security issues > > [1], it does not have great performance and there is really no > > development in there. > > > > Over the years, we still keep recommending people to use sftp > > instead, > > but its api is not that flexible and simple to be usable as a drop- > > in > > replacement in scripts nor for the occasional ad-hoc transfers of > > few > > files from one server to another. > > > > Before I start hacking, I would like to hear some opinions from > > others, > > whether this is something planned, welcomed or whether there are > > some > > good reasons to keep scp alive. > > > > I have in my mind three things/steps that would make it possible: > > > > * Update sftp client to be drop-in replacement for scp > > (and/or) > > * Change scp to use sftp internally > > > > * Modify sshd to use some compatibility "scpd" to support old > > clients > > > > and some time later > > > > * Remove scp or replace it with a symlink > > > > > > [1] http://www.openssh.com/txt/release-8.0 > > > > Any ideas/comments/suggestions? > > > > > > Best regards, > > -- > > Jakub Jelen > > Senior Software Engineer > > Security Technologies > > Red Hat, Inc. > > > > _______________________________________________ > > openssh-unix-dev mailing list > > openssh-unix-dev at mindrot.org > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >-- Jakub Jelen Senior Software Engineer Security Technologies Red Hat, Inc.
The fact that scp has been carried for ages is part of its value proposition. It?s going to work. Much as I am fond of my dd/cat/tar/smoke signal hacks, scp is the standard. A bit of a tangent: There is a security hole in scp, ssh, TLS, basically everything running over IP. As transfer sizes have increased significantly, the identity of a certain population of files being moved is increasingly obvious due to byte count, compressed or otherwise. No, you don?t get to pad every transfer out to the maximum possible size, or delay transfers until you can aggregate payloads. Yes, somebody thought of that. There are limits. On Tue, Jun 16, 2020 at 9:47 AM Jakub Jelen <jjelen at redhat.com> wrote:> Hello all, > > I believe we all can agree that scp is ugly protocol carried for ages > only for its simplicity of its usage and really no dependencies as it > is installed together with every ssh client. But as we have seen > recently, its simplicity and flexibility comes with security issues > [1], it does not have great performance and there is really no > development in there. > > Over the years, we still keep recommending people to use sftp instead, > but its api is not that flexible and simple to be usable as a drop-in > replacement in scripts nor for the occasional ad-hoc transfers of few > files from one server to another. > > Before I start hacking, I would like to hear some opinions from others, > whether this is something planned, welcomed or whether there are some > good reasons to keep scp alive. > > I have in my mind three things/steps that would make it possible: > > * Update sftp client to be drop-in replacement for scp > (and/or) > * Change scp to use sftp internally > > * Modify sshd to use some compatibility "scpd" to support old clients > > and some time later > > * Remove scp or replace it with a symlink > > > [1] http://www.openssh.com/txt/release-8.0 > > Any ideas/comments/suggestions? > > > Best regards, > -- > Jakub Jelen > Senior Software Engineer > Security Technologies > Red Hat, Inc. > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev >
The fundamental problem is that the command line syntax of sftp is not great. There's no fundamental reason that can't be fixed, without even changing the existing sftp protocol. But as long as scp (and rsync) exist the motivation to actually make the sftp command line tool nicer is so low that even though people have talked about it for 20 years, it hasn't happened yet. (And even though it would require orders of magnitude less effort than has been expended complaining about sftp & scp over the last 20 years...) I don't see an obvious way forward until someone decides to actually make the sftp client nicer to use. (No need to even talk about it: just show up with a finished version that does what scp does!)
Maybe Matching Threads
- Deprecation of scp protocol and improving sftp client
- Deprecation of scp protocol and improving sftp client
- Deprecation of scp protocol and improving sftp client
- Deprecation of scp protocol and improving sftp client
- Deprecation of scp protocol and improving sftp client