openssh bugs - Feb 2004 - [Bug 801] Improve usability of scp by adding a "make path" option

http://bugzilla.mindrot.org/show_bug.cgi?id=801

           Summary: Improve usability of scp by adding a "make path"
option
           Product: Portable OpenSSH
           Version: -current
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P3
         Component: scp
        AssignedTo: openssh-bugs at mindrot.org
        ReportedBy: sdturne at us.ibm.com


I would like to see an option added to scp that would cause the creation of any
non-existent directories leading up to the target file. For example if I run the
following command:

    scp -p file.dat remote:some/new/dir/file.dat

and any part of the "some/new/dir/" path does not exist, an error is
returned.
It would be very useful to have a new option, say "-m", that would
cause scp to
automatically create the missing directories without requiring a separate
"ssh
remote mkdir -p some/new/dir/" invocation.

I see this as a performance enhancement as well as a convenience.  I have
scenarios where I might have thousands of separate scp invocations, and it's
very expensive to run a separate invocation of "ssh remote mkdir -p"
to insure
the target path exists (not to mention the fact that the "mkdir -p"
command is
not portable to all the platforms I deal with).

I understand that scp is the follow-on to rcp, and that rcp has no such option.
 It has been suggested that this might be a "reason" NOT to implement
such a
feature in scp.   How disappointing to think that a replacement tool can never
improve on the original!  The GNU folks have never taken this approach, and as
long as the new feature doesn't break backward compatibility I can't see
the
logic in this argument.

I might also note that scp already has logic to create directories (used within
the -r logic), so perhaps the problem is already partially solved.

I hope this seems like a reasonable request, and thanks for your consideration.
Steve Turner



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801

mouring at eviladmin.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX



------- Additional Comments From mouring at eviladmin.org  2004-02-27 02:47
-------
     -p      Preserves modification times, access times, and modes from the
             original file.

Unable to use -p.   And scp is not a replacement for rcp.  It is a migration 
path.  Many places can move from rsh to ssh by just replacing one command.  The 
replacement for rcp is really 'sftp'.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From jmknoble at pobox.com  2004-02-27 03:18
-------
> I would like to see an option added to scp that would cause
> the creation of any non-existent directories leading up to the
> target file
Such functionality already exists, using ssh:

    cat file.dat |ssh remote \
    'mkdir -p some/new/dir && cd some/new/dir && cat
>file.dat'

This also works in reverse:

    (cd && mkdir -p some/new/dir && cd some/new/dir && \
    ssh remote 'cat file.dat' >file.dat)

This functionality can easily be automated via a script at either
end of the conversation.

Alternatively, if all you want to do is replicate directory
structures, use 'tar':

    tar cf - path/to/file.dat |ssh remote 'tar xvpf -'

or 'pax':

    pax -w path/to/file.dat |ssh remote 'pax -rvp e'

Good luck.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801

sdturne at us.ibm.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |



------- Additional Comments From sdturne at us.ibm.com  2004-02-27 03:53 -------
Ben, it does not appear that you understood my request.

I never suggested that "-p" be used to implement the "make
path" feature;
obviously that option is already used.  My suggestion was to use "-m".

Your statement that "scp is not a replacement for rcp" is simply not
correct;
the documentation even states that it is.

How could I use sftp instead of scp?  It doesn't operate anything close to
what
I need, and it doesn't preserve attributes as does "scp -p".


Jim,

Your first two suggestions don't preserve the permissions or timestamps of
the
source files; this is critical.

Your follow-on suggestions assume that my source file is already in the
directory hierachy that I need on the target; this is not the case.

There are most certainly umpteen other ways to "work around" my
problem, but
aren't they all ugly, slow and kludgy compared to a simple:

        scp -pm file.dat remote:some/new/dir/file.dat

?  As I mentioned in a parallel discussion in comp.security.ssh, I've seen
the
scp.c source code, and I can't envision this change requiring more than
about 25
or 50 lines of code...



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From smoogen at lanl.gov  2004-02-27 05:49 -------
It may only take 30 lines, but scp is 'deprecated' by OpenBSD and the
various
IETF  proposed standards I have seen. That means that no new options/additions
will be added to scp and all work should be on the recommended replacement,
sftp.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From sdturne at us.ibm.com  2004-02-27 07:22 -------
I have seen *zero* evidence of that, nor do I see any after searching the
OpenBSD website, www.ietf.org, or an exhaustive search of google.  Can you
supply any evidence that scp is in fact considered deprecated by OpenBSD (or
anyone else)?

I hope you'll pardon me for saying so, but attempting to make scp
"deprecated"
seems more than a bit ludicrous to me.  As wonderful as I'm sure sftp is (I
haven't used it to any great extent), it is *not* a replacement for scp.

Between this bug report and the comp.security.ssh discussion, the amount of
negative feedback I've gotten for such a small request really surprises me.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801

djm at mindrot.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|                            |WONTFIX



------- Additional Comments From djm at mindrot.org  2004-02-27 07:47 -------
You aren't listening: we aren't going to change scp.

Patches to teach sftp recursive operations and the few remaining bits of scp
commandline compatibility are welcome.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From dtucker at zip.com.au  2004-02-27 07:53 -------
The point is scp is a non-standard.  The closest thing it has to a
specification is "what rcp does".  The same command is used on both
ends
of the connection, and it has to interoperate with other implementations.

New features are more likely in sftp, since the protocol is standardized
(well, a draft standard), and the client and server are decoupled.  In
fact, you can already do what you want with the batch option, albeit in
a somewhat verbose way:

(echo -mkdir /tmp/dir; echo put -P file /tmp/dir/file) | sftp -b - localhost




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From markus at openbsd.org  2004-02-27 08:33 -------
there's no way to fix scp without breaking interoperbility.

better teach scp to speak sftp.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From sdturne at us.ibm.com  2004-02-27 09:41 -------
Damien Miller wrote:
>You aren't listening: we aren't going to change scp.
Really?  I'm the one who's not listening?  That's funny I kinda got
the idea
that most of you weren't *reading*.  I'm listening just fine, I'm
just not
willing to let the issue drop without a decent explanation.  Yours wasn't
it.
Both you and Ben closed my request without sufficient explanation and without
the slightest shred of courtesy.  You really should try working directly with
paying customers some time; you'd both do real well.

<deep breath>

Darren,

Your explanations are the first I've seen all day that hit the nail on the
head,
and now I understand what the real danger is in changing scp.  I'm now quite
happy to accept the fact that this issue won't be addressed, and thank you
so
much for clearing it up.  I applaud you sir.

I also want to apologize if some of my responses seem short-fused, but this
topic has been getting some snide remarks over in comp.security.ssh as well, and
I've been carrying some of that frustration over to this discussion.  Sorry
'bout that.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From mouring at eviladmin.org  2004-02-27 10:01
-------
You must understand this type of thing comes up every 4 - 5 months, and after a 
while it gets tiresome when people don't alway think through the changes
they
are trying to push.

Examples from Bugzilla:

http://bugzilla.mindrot.org/show_bug.cgi?id=424
http://bugzilla.mindrot.org/show_bug.cgi?id=485
http://bugzilla.mindrot.org/show_bug.cgi?id=619
http://bugzilla.mindrot.org/show_bug.cgi?id=431

I'm sure if I dug through the OpenSSH-Unix-Dev@ list I could pull more
examples.

<shrug> Others have already explained this.  I'm going back to my day
job.=)

- Ben




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801





------- Additional Comments From dtucker at zip.com.au  2004-02-27 10:19 -------
Created an attachment (id=553)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=553&action=view)
Add scp to FAQ

Since this seems to be a FAQ :-)



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=801

djm at mindrot.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
    Attachment #553|                            |ok
             Status|                            |



------- Additional Comments From djm at mindrot.org  2004-02-27 10:41 -------
(From update of attachment 553)
ok




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.