rsync - Mar 2008 - New rsync option checksum-path

This is a suggestion for adding to rsync an option
"--checksum-path=PROGRAM".

Once this option is given, provided it is accompained by option
"--checksum", both the receiving and sending side of rsync would
compute checksums calling PROGRAM, instead of the usual checksum
program.

Why is this usefull?  Suppose I have two collections of MP3 files at
different locations which I want to upate.  Suppose also that for some
reason the testing of modtime is not efficient for distinguishing
changeg files.  Recall that changes to ID3 tags inside MP3 files do
not change filesizes, and hence neither filesize is a good criteria
for identifying changes.

Suspecting that the changes were made only to ID3 tags, a very common
situation, one could write a shell script, say SHORTSUM, which would
calculate a checksum only on the first 10K bytes, for example, where
ID3 tags reside.

Using

  rsync -av --checksum-path=SHORSUM --checksum /LOCAL-MP3-DIRECTORY/
SERVER:/REMOTE-MP3-DIRECTORY/

would therefore save a lot of processing power.

I am aware that rsync was designed in a way to dispense with previous
knowledge of how files are structured.  However, a simple change as
proposed would allow users to custumize rsync based on such knowledge.

Sincerely, Ruy Exel

On Mon, 2008-03-10 at 13:37 -0300, Ruy Exel wrote:
> This is a suggestion for adding to rsync an option
"--checksum-path=PROGRAM".
> 
> Once this option is given, provided it is accompained by option
> "--checksum", both the receiving and sending side of rsync would
> compute checksums calling PROGRAM, instead of the usual checksum
> program.
I don't think this option would be useful to enough people to justify
including it in the official rsync (of course, the final decision is
Wayne's), but by all means make your own version of rsync with the
option if it does what you need.  Another approach would be to write a
script that runs the custom checksum program on the source and
destination files, determines which source files need to be copied, and
passes the list to rsync with --files-from .

Matt

On Mon, Mar 10, 2008 at 01:37:25PM -0300, Ruy Exel
wrote:
> Suspecting that the changes were made only to ID3 tags, a very common
> situation, one could write a shell script, say SHORTSUM, which would
> calculate a checksum only on the first 10K bytes, for example, where
> ID3 tags reside.
One thing you can do is to use the existing patch, checksum-reading,
which allows the exchange of pre-computed checksum information that
is matched against a more strict set of stat values than the normal
"quick check" (i.e. it includes ctime and inode).  You could modify
the included perl script (support/rsyncsums) so that it generates a
checksum using an alternate means than the normal full-file checksum
(which is cached as an MD4 and an MD5 checksum for both old and new
protocols -- so just change both values to the same alternate sum).
As long as you regularly run the rsyncsums script on both systems
(and make sure that both the sender and the receiver rsync get the
--sumfiles=strict option), then rsync will figure out which files
have really changed without having to read all the files.  (Even the
rsyncsums script run is efficient, as it just updates files where
the mtime/ctime/size/inode have changed since the last cache.)

Note that the above solution doesn't require that you alter the
checksum method in order to make things fast, so you could just
leave it alone if you are willing to make regular runs of the
rsyncsums script.

You do need to be careful not to use the rsyncsums script on files
that are being actively written, as there is a small window where a
race condition can bite you.  To trigger it, rsync needs to stat a
file with the same mtime and ctime as the current time, read the
file, re-stat the file and discover that the stat info is unchanged
(including no change in size), and also have a concurrent writer
that changes the file's data during the period in between when rsync
reads the file but before the end of the current second, with no
further changes being made to the file after the current second
ends.  The rsyncsums script could be changed to not cache the
checksum of a file which has a "now" ctime value, which would
avoid this, but it does not do that at the moment.

..wayne..