User demand and it's a deliverable for a project. Like I said in my
first message I know that rsync can do basically the same thing but ya
know, deliverables need to be delivered. Just thought I'd share the
results.
On 4/1/21 3:00 PM, Ron Frederick wrote:> If you?re looking for incremental copy capabilities, why not just use
rsync, which already runs nicely over SSH?
>
> On Apr 1, 2021, at 10:50 AM, rapier <rapier at psc.edu> wrote:
>> Howdy all,
>>
>> I know development on SCP is discouraged but being that it's still
in wide use I thought I would do some work some of my users have been asking for
and allow SCP to resume from a partial transfer.
>>
>> When enabled the resume function will compare the source and target
files. The source host will and target host will pass MD5 hashes back and forth
in order to compare the hash of the file fragment on the target and the file up
to the length of the source file. If these hashes match the source will seek to
the appropriate byte position and start sending the rest of the file. This will
then be appended to the target.
>>
>> Example:
>> An scp transfer of a 128,000 byte file terminates after transferring
64,000 bytes. The user then issues the SCP command again with the -R (resume)
option. The hash of the 64k fragment on the target is compared to the hash of
the source file up to 64k. The hashes match so the source seeks to the 64,001th
byte and then starts send the remainder of the file. This is written to a temp
file on the target which then appends it to the target after the end of the
transfer.
>>
>> If the hash doesn't match then then entire source file is copied to
the target and replaces the fragment.
>>
>> In the event that the source and target files have the same size and
same hash then the file is skipped (no transfer of data). This end sup being
handy if the user is doing a recursive transfer or using a wildcard.
>>
>> Both sides have to be using a resume enabled SCP for this to work.
Since scp uses the first scp in the users path I've added an option that
lets users define which scp to use on the remote side. This can be set with
'-s' e.g. 'scp -R -s /opt/openssh/bin/scp' foo* you at
host:~/foofiles/'
>>
>> The code is available at https://github.com/rapier1/openssh-portable in
the resume-scp branch. The patch file is ~1k lines long so I didn't think
posting it here would be the right move.
>>
>> Note: This is the first functional implementation of this so there are
a lot of debug statement and I am sure there are ways of improving this. I also
know that users can just use rsync. That said, this was a deliverable and the
users wanted this. So I did it. Anyway, I'm open to any and all suggestions,
modifications, criticism and so forth.
>>
>> Thanks,
>>
>> Chris
>