Folks-
Just wanted to plug a totally awesome software package from a group I
know: UDR (UDT Enabled Rsync).
For those not familiar with UDT, it is a low level network protocol
based on UDP that allows for high speed transfers over high latency WAN
networks:
http://udt.sourceforge.net/
For a while the UDT API was available, developed by the Laboratory for
Advanced Computing at the University of Chicago, but there were little
development around it for actually developing a software suite to allow
for high speed WAN transfers, such as can be achieved by Aspera,
GridFTP, FDT, and a couple others. The problem with those often is:
Aspera: Great but *crazy* expensive
GridFTP: Not bad but non-trivial to set up
FDT: Java (Eh...)
But the awesome thing here is that UDR is a lightweight wrapper for
rsync that allows for rsync functionality, but uses UDT as the
underlying protocol for transfer (no rsh or ssh). It authenticates over
ssh and then transfers the data over UDT streams. And supports encryption.
Right now it is kind of in beta but we've been using it for a while and
it's very stable. It has been tested on Linux, BSD and OSX. It may
compile on other platforms but not much testing has been done on those.
Written in C++.
https://github.com/LabAdvComp/UDR
You can clone the repo with 'git clone' and build the code. It compiles
very easily and only requires the OpenSSL library as a dependency.
As an example: We have been trying to transfer data from California to
our servers in Germany for a while, and have only be getting 10Mb/s.
With UDR we get 700Mb/s. Not bad.
There are details on the GitHub page. Check it out!!
-erich