openssh unix dev - May 2020 - Parallel transfers with sftp (call for testing / advice)

Le 10/04/2020 ? 01:55, Darren Tucker a ?crit?:
> On Thu, 9 Apr 2020 at 01:34, Cyril Servant <cyril.servant at
gmail.com> wrote:
> [...]
>> Each of our front
>> nodes has an outgoing bandwidth limit (let's say 1Gb/s each,
generally more
>> limited by the CPU than by the network bandwidth),
> You might also want to experiment with the Ciphers and MACs since
> these can make a significant difference in CPU utilization and, if
> that's the bottleneck, your throughput.  Which one is best will vary
> depending on your hardware, but it's likely to be either AES GCM if
> the hardware has AES instructions or chacha20-poly1305 if not.
>
> In the first example below the bottleneck is the source's relatively
> elderly 2.66GHz Intel CPU.  In the second it's the gigabit network
> between them.
>
> $ scp -c aes256-ctr -o macs=hmac-sha2-512
> ubuntu-18.10-desktop-amd64.iso.bz2 nuc:/tmp/
> ubuntu-18.10-desktop-amd64.iso.bz2            100% 1899MB  63.5MB/s   00:29
>
> $ scp -c chacha20-poly1305 at openssh.com
> ubuntu-18.10-desktop-amd64.iso.bz2 nuc:/tmp/
> ubuntu-18.10-desktop-amd64.iso.bz2            100% 1899MB 112.1MB/s   00:16
>
Hi,

As Cyril said, we are aware of the cpubound aspect of the available 
ciphers and MACs in OpenSSH and have already selected the most efficient 
one for our transfers after several benchmarking sessions.

Current hardware processors have a limited core capacity. Core 
frequencies are staying roughly at the same level since many years now 
and only core count are increasing, relying on developpers to play with 
parallelism in order to increase the compute throughput. The future does 
not seem brighter in that area.

In the meantime, network bandwidth has still increased at a regular 
pace. As a result, a cpu frequency that was once sufficient to fill the 
network pipe is now only at a fraction of what the network can really 
deliver. 10GE ethernet cards are common nowadays on datacenter servers 
and no openssh ciphers and MACs can deliver the available bandwidth for 
single transfers.

Introducing parallelism is thus necessary to leverage what the network 
hardware can offer.

The change proposed by Cyril in sftp is a very pragmatic approach to 
deal with parallelism at the file transfer level. It leverages the 
already existing sftp protocol and its capability to write/read file 
content at specified offsets. This enables to speed up sftp transfers 
significantly by parallelizing the SSH channels used for large 
transfers. This improvement is performed only by modifying the sftp 
client, which is a very small modification compared to the openssh 
codebase. The modification is not too complicated to review and validate 
(I did it) and does not change the default behavior of the cli.

It exists tools that offers parallel transfers of large files but we do 
really want to use OpenSSH for that purpose because it is the only 
application that we can really trust (by the way, thank you for making 
that possible). I do no think that we are the only one to think like 
this and I am pretty sure that such a change in the main code base of 
OpenSSH would really help users to use their hardware more efficiently 
in various situations.

Best regards,

Matthieu

Matthieu Hautreux wrote:
> The change proposed by Cyril in sftp is a very pragmatic approach to 
> deal with parallelism at the file transfer level. It leverages the 
> already existing sftp protocol and its capability to write/read file 
> content at specified offsets. This enables to speed up sftp transfers 
> significantly by parallelizing the SSH channels used for large 
> transfers. This improvement is performed only by modifying the sftp 
> client, which is a very small modification compared to the openssh 
> codebase. The modification is not too complicated to review and validate 
> (I did it) and does not change the default behavior of the cli.
I think you make a compelling argument. I admit that I haven't
reviewed the patch, even though that is what matters the most.

I guess that noone really minds ways to make SFTP scale, but ever since
the patch was proposed I have been thinking that the paralell channel
approach is likely to introduce a whole load of not very clean error
conditions regarding reassembly, which need to be handled sensibly both
within the sftp client and on the interface to outside/calling processes.
Can you or Cyril say something about this?

And another thought - if the proposed patch and/or method indeed will not
go anywhere, would it still be helpful for you if the sftp client would
only expose the file offset functionality? That way, the complexity of
reassembly and the associated error handling doesn't enter into OpenSSH.


Kind regards

//Peter

Peter Stuge wrote:
> 
> Matthieu Hautreux wrote:
>> The change proposed by Cyril in sftp is a very pragmatic approach to 
>> deal with parallelism at the file transfer level. It leverages the 
>> already existing sftp protocol and its capability to write/read file 
>> content at specified offsets. This enables to speed up sftp transfers 
>> significantly by parallelizing the SSH channels used for large 
>> transfers. This improvement is performed only by modifying the sftp 
>> client, which is a very small modification compared to the openssh 
>> codebase. The modification is not too complicated to review and
validate
>> (I did it) and does not change the default behavior of the cli.
> 
> I think you make a compelling argument. I admit that I haven't
> reviewed the patch, even though that is what matters the most.
If you want to review the code, here is a direct link to the patch:
https://github.com/openssh/openssh-portable/compare/V_8_2_P1...cea-hpc:V_8_2_P1_PSFTP1
> I guess that noone really minds ways to make SFTP scale, but ever since
> the patch was proposed I have been thinking that the paralell channel
> approach is likely to introduce a whole load of not very clean error
> conditions regarding reassembly, which need to be handled sensibly both
> within the sftp client and on the interface to outside/calling processes.
> Can you or Cyril say something about this?
Indeed, reassembly must be handled properly. Depending on the filesystem
you're
writing to, the block size can vary. Having a "power of 2 bytes" chunk
is the
starting point. Performance-wise, our storage experts asked us to use 2MB or
more chunks. I chose the arbitrary value of 64MB chunks because I don't see
any
advantage of using smaller chunks. After all, we're talking about using
parallel transfers on high bandwidth links. Of course this value can be
discussed and easily changed (it's the base_chunk_size variable). In order
to
be more exact, a chunk size will be a multiple of base_chunk_size, so a 4GB
file transferred with 4 channels will be cut in 4 1GB chunks.

The main source of errors during reassembly is if the copy_buffer_len (-B
option) is set to a "non power of 2" value. This will lead to writes
sitting
partially on 2 blocks, and probably corrupt the file. Writing simultaneously
the start and the end of a block on 2 different NFS clients is a really bad
idea. That's why I issue a warning if -n > 0 and -B is not a power of 2.

Concerning error handling within threads, if a thread encounters a blocking
error, the other threads will end their current chunk copy, then stop doing
anything.
> And another thought - if the proposed patch and/or method indeed will not
> go anywhere, would it still be helpful for you if the sftp client would
> only expose the file offset functionality? That way, the complexity of
> reassembly and the associated error handling doesn't enter into
OpenSSH.
There is no server side change in this patch, so I don't think we can talk
about conplexity of reassembly. Once base_chunk_size is set and a warning or an
error is raised if copy_buffer_len is not a power of 2, there is nothing more
than during a reput or a reget.

Of course, exposing the file offset functionality would help creating a new
software on top of the sftp client, but at the cost of simplicity. And if you
do this, in the case of "get", the new software will need to send a
lot of "ls"
commands in order to be aware of the directory structure, the file sizes in
order to correctly split transfers? I feel like it's reinventing the wheel.
-- 
Cyril

On Tue, May 5, 2020 at 4:31 AM Peter Stuge <peter at stuge.se>
wrote:
>
> Matthieu Hautreux wrote:
> > The change proposed by Cyril in sftp is a very pragmatic approach to
> > deal with parallelism at the file transfer level. It leverages the
> > already existing sftp protocol and its capability to write/read file
> > content at specified offsets. This enables to speed up sftp transfers
> > significantly by parallelizing the SSH channels used for large
> > transfers. This improvement is performed only by modifying the sftp
> > client, which is a very small modification compared to the openssh
> > codebase. The modification is not too complicated to review and
validate
> > (I did it) and does not change the default behavior of the cli.
>
> I think you make a compelling argument. I admit that I haven't
> reviewed the patch, even though that is what matters the most.
>
> I guess that noone really minds ways to make SFTP scale, but ever since
> the patch was proposed I have been thinking that the paralell channel
> approach is likely to introduce a whole load of not very clean error
> conditions regarding reassembly, which need to be handled sensibly both
> within the sftp client and on the interface to outside/calling processes.
> Can you or Cyril say something about this?
I find it an unnecessary feature given the possibilities of
out-of-band parallelism with multiple scp sessions transmitting
diferent manifests of files, of sftp to do the same thing, and of
tools like rsync to do it more efficiently by avoiding replication of
previously transmitted data and re-connection to complete partial
transmisions. It sounds like a bad case of "here, let me do this at a
different level of the stack" that is not normally necessary and has
already been done more completely and efficiently by other tools.
> And another thought - if the proposed patch and/or method indeed will not
> go anywhere, would it still be helpful for you if the sftp client would
> only expose the file offset functionality? That way, the complexity of
> reassembly and the associated error handling doesn't enter into
OpenSSH.
Re-assembly, eror handling, and delivery verification were done by
rsync ages ago. It really seems like re-inventing the wheel.