search for: builk

Hello, I'd like to share with you an evolution I made on sftp. 1. The need I'm working at CEA (Commissariat ? l'?nergie atomique et aux ?nergies alternatives) in France. We have a compute cluster complex, and our customers regularly need to transfer big files from and to the cluster. Each of our front nodes has an outgoing bandwidth limit (let's say 1Gb/s each, generally more

...2020 at 11:31 AM Cyril Servant <cyril.servant at gmail.com> wrote: >> >> Hello, I'd like to share with you an evolution I made on sftp. > > It *sounds* like you should be using rparallelized rsync over xargs. > Partial sftp or scp transfers are almost inevitable in builk transfers > over a crowded network, and sftp does not have good support for > "mirroring", only for copying content. > > See https://stackoverflow.com/questions/24058544/speed-up-rsync-with-simultaneous-concurrent-file-transfers This solution is perfect for parallel sending...