I'm watching my backup via rsync, throttled to a very low speed. Looks like downloading apache logs is taking the longest time (when I'm rsyncing over an old copy of the same data) because it's not noticing that, for example, www.chaosreigns.com-access.log.196.gz on the origin is the same file as www.chaosreigns.com-access.log.186.gz on the destination, so *everything* is getting unnecessarily retransmitted. How do people deal with that? In this case bandwidth is basically all that matters (I'm backing up 7.5gb of data with --bwlimit=8), time and cpu are not very important. So I was wondering if it was possible to tell rsync to pay more attention to checksums than filenames / paths, so it can notice what the above file got renamed to and mimic the rename without a retransmit? Man page is looking like no, any chance of that getting implemented? Or is checksumming two copies of 7.5gb of data not something I actually want to do (p3 550 -> A64 x2 3800)? The other obvious option would be to use dates when renaming logs instead of incrementing log numbers. I'm kind of surprised logrotate doesn't seem to support that. Wouldn't be too hard to write a script to do it pretty cleanly. Anything I'm missing? -- "For gasoline vapor, the explosive range is from 1.3 to 6.0% vapor to air...useful against soft targets such as...armored vehicles...and bunkers." - http://www.fas.org/man/dod-101/sys/dumb/fae.htm http://www.ChaosReigns.com
On Tue, Jan 09, 2007 at 04:02:14PM -0500, Darxus@ChaosReigns.com wrote:> it's not noticing that, for > example, www.chaosreigns.com-access.log.196.gz on the origin is the same > file as www.chaosreigns.com-access.log.186.gz on the destination, so > *everything* is getting unnecessarily retransmitted.The --fuzzy option might help, but only if the filenames that moved don't already exist. Rsync expects that an existing file is the right file to use for old data, and never checks for a different file as a better source of old data. For the case of rotated logs, you're better off first running an appropriate number of file rotations on the receiving system, and then doing a copy of the new files. ..wayne..
Apparently Analagous Threads
- --link-dest isn't linking
- [Bug 55596] New: weston-launch causes graphical corruption / lockup
- [Bug 10233] New: rsync is spending a lot of time lstat64()'ing --exclude'd files
- rsync -vae ssh user@host1:/tmp/dir user@host2:/tmp/
- [Bug 48074] New: 3D / OpenGL clients used with wayland always cause an immediate hang