openssh unix dev - Apr 2021 - SCP with Resume Feature

On Tue, 6 Apr 2021, rapier wrote:
> On 4/6/21 10:04 PM, Damien Miller wrote:
> > On Tue, 6 Apr 2021, rapier wrote:
> > 
> > > Looking at the performance - on my systems sftp seems to be a bit
slower
> > > than scp when dealing with a lot of small files. Not sure why
this is
> > > the case as I haven't looked at the sftp code in years.
> > 
> > the OpenSSH sftp client doesn't do inter-file pipelining - it only
> > pipelines read/writes within a transfer, so each new file causes a
> > stall.
> > 
> > This is all completely fixable on the client side, and shouldn't
apply
> > to things like sshfs at all.
> 
> Gotcha. Is this because of how it sequentially loops through the readdirs
in
> two _dir_internal functions?
Only partly - the client will do SSH2_FXP_READDIR to get the full list of
files and then transfer each file separately. The SSH2_FXP_READDIR are not
pipelined at all, there is no pipelining between obtaining the file list
and the file transfers. Finally each file transfer incurrs a pipeline
stall upon completion.
> If so I'm wondering if you could spawn per file
> threads to get some concurrency within a directory.
I don't think we want a threaded sftp client and AFAIK it isn't
necessary
for the main problem. We could add inter-operation pipelining by adding a
work queue structure and driving the next operation from that. This is
similar to what happens inside do_download()/do_upload() already, but
extended to persist across and between different operations.
> Just curious and this is
> the first time I've looked at the sftp code in years. I hope you
don't mind
> the questions.
Not at all :)

-d

On Apr 6, 2021, at 10:47 PM, Damien Miller <djm at mindrot.org>
wrote:
> On Tue, 6 Apr 2021, rapier wrote:
>> On 4/6/21 10:04 PM, Damien Miller wrote:
>>> On Tue, 6 Apr 2021, rapier wrote:
>>> 
>>>> Looking at the performance - on my systems sftp seems to be a
bit slower
>>>> than scp when dealing with a lot of small files. Not sure why
this is
>>>> the case as I haven't looked at the sftp code in years.
>>> 
>>> the OpenSSH sftp client doesn't do inter-file pipelining - it
only
>>> pipelines read/writes within a transfer, so each new file causes a
>>> stall.
>>> 
>>> This is all completely fixable on the client side, and
shouldn't apply
>>> to things like sshfs at all.
>> 
>> Gotcha. Is this because of how it sequentially loops through the
readdirs in
>> two _dir_internal functions?
> 
> Only partly - the client will do SSH2_FXP_READDIR to get the full list of
> files and then transfer each file separately. The SSH2_FXP_READDIR are not
> pipelined at all, there is no pipelining between obtaining the file list
> and the file transfers. Finally each file transfer incurrs a pipeline
> stall upon completion.
The good news here is that from a protocol standpoint a server can already break
up a READDIR response into multiple chunks. So, while there will still be a
stall between READDIR calls on a directory with a very large number of files, a
client can start to pipeline the transfers of those files or recursive READDIR
calls for subdirectories without waiting for the entire listing of files in the
parent directory to be returned, once there?s some mechanism in place to manage
that work. To avoid overwhelming the server, you?ll probably want to put a cap
on the number of simultaneous requests to any given server, but that can all be
managed in the client.

In AsyncSSH, I implemented a scandir() call that returns an async iterator of
remote directory entries from READDIR that begins to return results even before
the full list of file names in a directory has been returned and then used that
to implement an rmtree() call on the client which parallelized recursive
deletion of a remote directory tree and saw a significant speedup on trees with
a large number of files/subdirectories. I haven?t yet updated my recursive file
transfer client code to leverage this since there was already a good amount of
parallelism on the transfers themselves, but perhaps I?ll look into doing this
next. With a large number of very small files, I would expect to see some
benefit from that.

That said, is the SCP implementation in OpenSSH currently doing any file-level
parallelization? I wouldn?t expect it to, so I?m not sure that would explain the
performance difference. If I had to guess, it?s more likely due to the fact that
there?s a single round-trip with SCP for each file transfer, whereas SFTP
involves separate requests to do an open(), read(), stat(), etc. each of which
has its own round-trip. Some of those (such as the read() calls) are
parallelized, but you still have to pay for the open() before beginning the
reads, and possibly for other things like stat() when preserving attributes.
-- 
Ron Frederick
ronf at timeheart.net