Hi,
I am hoping that rsync may be able to improve an existing network backup
I've got:
tar czf - --files-from $FILE_LIST | ssh -i $AUTH -l $USER $HOST "cat >
${DEST}/${SOURCE}_${FILE}.tgz"
You can see that this backup uses tar to create a single compressed archive and
store it on a remote host using ssh. It occurred to me that it might be possible
to use rsync. I'm hoping rsync could be told to capture STDIN and treat it
as a single file that it would then sync with the named file at the remote end.
Perhaps something like this:
tar czf - --files-from $FILE_LIST | rsync - $HOST:${BACKUP_FILE}
I've been unable to find any reference to rsync being used in this fashion.
Can anyone tell me whether this is possible today, and if not whether it might
be considered for a future version?
Cheers,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20091119/f62fed05/attachment.html>
On Thu, Nov 19, 2009 at 08:14, Mark Young <mark_young at hotmail.com> wrote:> I've been unable to find any reference to rsync being used in this fashion. > Can anyone tell me whether this is possible today, and if not whether it > might be considered for a future version?In client mode, 'rsync' can accept a list of files to include/exclude on STDIN. In daemon mode, it uses STDIN to interact with Inetd. But it doesn't support a usage like you're describing. In other words, 'rsync' works like a replacement for 'scp', not 'ssh' like you're doing, here. But depending on your requirements, 'rsync' might not have any improvements to offer you. Unless a significant portion of the file data already exists on the remote side, 'rsync' will actually be *slower* than using 'tar' plus 'ssh'. If you actually want to take advantage of rsync, maybe you can explain a little more about your requirements, and people can make suggestions. I would ask two questions: - Do you absolutely need to have separate tarballs on the remote side, per backup? - Do the remote-side backups absolutely need to be compressed? -Ryan
On Thu, 2009-11-19 at 13:14 +0000, Mark Young wrote:> I'm hoping rsync could be told to capture STDIN and treat it as a > single file that it would then sync with the named file at the remote > end. Perhaps something like this: > > tar czf - --files-from $FILE_LIST | rsync - $HOST:${BACKUP_FILE}This could probably be implemented as a simple extension of the current --copy-devices functionality. My only concern is that if the delta-transfer algorithm makes a mistake, rsync will want to try the transfer again, but stdin can only be consumed once. This could be fixed by hacking in support for forking a subprocess to generate the source content, or by using as the source a FUSE-based filesystem that exposes the virtual tar file. Another option would be to use rdiff(1). It can read from stdin, and I don't think it attempts to retry if it makes a mistake. To expand upon Ryan's remarks, even if you have an old compressed tarball on the destination, the delta-transfer algorithm might not find any matches since compression introduces non-local effects. To solve that, the "gzip-rsyncable" patch (to gzip) would be needed. It was maintained with rsync for a while but was apparently discontinued around 3.0.0pre6. -- Matt