rsync - Jul 2010 - --compare-dest weirdness

Hi All,

I am writing a backup program for my computer. brief outline is as follows.

Running ubuntu 10.04
2 main partitions, / and /home, both ext3. 1 external usb hdd, ext3,
mounted to /backups/main.
once every couple of days, rsync backs up, using following command,
everything worth backing up in / and /home partitions to a folder
/backups/main/Full. command: "rsync -vrhRupElog --delete-during
--delete-excluded --exclude-from=/backups/.config/full.exclude
--log-file=/backups/.logs/full.rlog / /backups/main/Full/". this works
perfectly

now what i want to do, is every few hours, rsync ONLY the files
changed in the last few days to a folder in
/backups/main/Incremental/YYYYMMddTHHmm, where the yyyy... is a string
representing the date and time. (and then tar.gz the previous
directory to save space) the command i am using for this is : "rsync
-vrhRupElog --delete-during --delete-excluded
--exclude-from=/backups/.config/incremental.exclude
--log-file=/backups/.logs/incremental.rlog
--compare-dest=/backups/main/Full/ /
/backups/main/Incremental/Full/20100716T1355" the incremental.exclude
differs from the full.exclude only in that i also exclude .mp3, .avi
and other large and non-critical file types.

the problem is that rsync is behaving as though it is ignoring the
compare-dest option. it throws no error about this dir not existing,
and i have tried giving it as a relative dir to the destination with
same result.

interestingly, i tried to see if something was wrong with my statments by doing:
mkdir ~/rsynctest/dir1
mkdir ~/rsynctest/dir2
mkdir ~/rsynctest/dir3

nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)

cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/

checked md5sums of both files in both dirs to ensure they were identical

the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
/home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/

the two files from dir1 were copied to dir3, even though identical
copies, with the same names, existed in dir2.

i also tried this again with the compare dir relative to the dest dir,
i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
/home/kevin/rsynctest/dir3/ again with same result

is this a bug, or have i got the wrong end of the stick regarding --compare-dir?

thanks all.

Regards
Kevin Murray
-------------- next part --------------
A non-text attachment was scrubbed...
Name: full.exclude
Type: application/octet-stream
Size: 2465 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20100716/aae7b0ca/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: incremental.exclude
Type: application/octet-stream
Size: 4063 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20100716/aae7b0ca/attachment-0001.obj>

> Hi All,
> 
> I am writing a backup program for my computer. brief outline is as follows.
> 
> Running ubuntu 10.04
> 2 main partitions, / and /home, both ext3. 1 external usb hdd, ext3,
> mounted to /backups/main.
> once every couple of days, rsync backs up, using following command,
> everything worth backing up in / and /home partitions to a folder
> /backups/main/Full. command: "rsync -vrhRupElog --delete-during
> --delete-excluded --exclude-from=/backups/.config/full.exclude
> --log-file=/backups/.logs/full.rlog / /backups/main/Full/". this works
> perfectly
> 
> now what i want to do, is every few hours, rsync ONLY the files
> changed in the last few days to a folder in
> /backups/main/Incremental/YYYYMMddTHHmm, where the yyyy... is a string
> representing the date and time. (and then tar.gz the previous
> directory to save space) the command i am using for this is : "rsync
> -vrhRupElog --delete-during --delete-excluded
> --exclude-from=/backups/.config/incremental.exclude
> --log-file=/backups/.logs/incremental.rlog
> --compare-dest=/backups/main/Full/ /
> /backups/main/Incremental/Full/20100716T1355" the incremental.exclude
> differs from the full.exclude only in that i also exclude .mp3, .avi
> and other large and non-critical file types.
> 
> the problem is that rsync is behaving as though it is ignoring the
> compare-dest option. it throws no error about this dir not existing,
> and i have tried giving it as a relative dir to the destination with
> same result.
> 
> interestingly, i tried to see if something was wrong with my statments by
doing:
> mkdir ~/rsynctest/dir1
> mkdir ~/rsynctest/dir2
> mkdir ~/rsynctest/dir3
> 
> nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
> nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)
> 
> cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/
> 
> checked md5sums of both files in both dirs to ensure they were identical
> 
> the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
> /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/
> 
> the two files from dir1 were copied to dir3, even though identical
> copies, with the same names, existed in dir2.
> 
> i also tried this again with the compare dir relative to the dest dir,
> i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
> /home/kevin/rsynctest/dir3/ again with same result
> 
> is this a bug, or have i got the wrong end of the stick regarding
--compare-dir?
> 
Hopefully, someone on the list will be able to provide you with some further
details regarding what is happening.

In the mean time I suggest that you consider looking at rsnapshot, rdiffbackup,
link-backup or LBackup.

All of the projects listed above and many more are listed at the following URL :
http://www.lbackup.org/alternatives

Certainly, with LBackup (disclaimer - I am a developer on this project) you are
able to configure the backup to have just one or two hard linked rotations by
specifying the following in the backup configuration file :

numRotations=1 # only the most recent successful backup will be kept
numRotations=2 # only the two most recent successful backups will be kept

LBackup will save space even if you save 50, 500 or 1000 backups due to hard
linking files which have not been changed or moved. The one issue is that it is
important not to exceed the your file systems inode limits.

With regards the creation of your compressed tar you could add a post action
into the resources/post-actions directory of your configuration directory and
create a compressed tar of the most recent backup. If you wanted to you could
even rsync this backup to one or more remote systems. The post action sub system
of each LBackup configuration directory makes this kind of customized setup
trivial. For example there is post action script bundled with LBackup which will
generate these dated linked directories within your backup set. The example
script is called : BACKUP-300-Generate-Time-Based-Backup-Links-Directory.bash

This script makes altering the format of the links, having multiple  formats or
different formats for different a snap.

If you are quite sure you want to roll your own rsync based backup then that is
fantastic.

The only suggestions I have with regards why your current system is not working
is that perhaps trying the --link-dest rather than the --compare-dest option may
provide some further insight into what is not working. However, I see no reason
why one would work while the other would not.

I also suggest that you try to check why the files are being updated and not
matched by adding the --itemize-changes option. This option should provide some
further details. In addition, you may want to also add some additional verbosity
by adding -vvv as an option. You currently have just one -v.

Are you running rsync as root or as a non-privilged user. I assume you are
running as root because you are using the --owner and --group options. Would you
confirm that this is the case.

Also I wonder if the --compare-dest option would work if you removed the
--relative option? It could be that option is causing some sort of problem with
regards the --compare-dest option?

Finally, ensure that permissions and time stamps are being correctly preserved
on destination. If the permissions are not being preserved, then this may be
causing the problems with --compare-dest which you have outlined.

I hope this helps and if not then hopefully someone else will chime in to
provide you with some further ideas and tips or maybe spot something which I
have missed.


---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU GPL
http://www.lbackup.org

> interestingly, i tried to see if something was wrong with my statments by
doing:
> mkdir ~/rsynctest/dir1
> mkdir ~/rsynctest/dir2
> mkdir ~/rsynctest/dir3
> 
> nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
> nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)
> 
> cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/
> 
> checked md5sums of both files in both dirs to ensure they were identical
> 
> the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
> /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/
> 
> the two files from dir1 were copied to dir3, even though identical
> copies, with the same names, existed in dir2.
> 
> i also tried this again with the compare dir relative to the dest dir,
> i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
> /home/kevin/rsynctest/dir3/ again with same result


I thought about this further and I think the problem you are having is that the
modification times are not being preserved and you are not using the --checksum
option.

As such, I suggest that you add the --times or --checksum option and see if that
helps.

I suspect that this will solve your problem. Let me know =:^)


---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU GPL
http://www.lbackup.org

Hi Henri

Thanks very much. LBackup looks good, but i need a solution which only
copies files which have changed, and does not link the others, as the
folder is destined for a tar archive. Also, the copied files must be
fully functional, i.e. must be whole files, not just differences like
rdiff. this is why im using rsync.

Using checksum works, however with over 600gb to backup it is not a
viable option.

the -i shows that most are being copied due to time differences, so in
theory -t should work? This does in fact work on a little test setup
on my work laptop, i will test it properly when i get home tonight.

Thanks

Regards
Kevin Murray



On 20 July 2010 12:40, Henri Shustak <henri.shustak at gmail.com>
wrote:
>> interestingly, i tried to see if something was wrong with my statments
by doing:
>> mkdir ~/rsynctest/dir1
>> mkdir ~/rsynctest/dir2
>> mkdir ~/rsynctest/dir3
>>
>> nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
>> nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)
>>
>> cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/
>>
>> checked md5sums of both files in both dirs to ensure they were
identical
>>
>> the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
>> /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/
>>
>> the two files from dir1 were copied to dir3, even though identical
>> copies, with the same names, existed in dir2.
>>
>> i also tried this again with the compare dir relative to the dest dir,
>> i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
>> /home/kevin/rsynctest/dir3/ again with same result
>
>
>
> I thought about this further and I think the problem you are having is that
the modification times are not being preserved and you are not using the
--checksum option.
>
> As such, I suggest that you add the --times or --checksum option and see if
that helps.
>
> I suspect that this will solve your problem. Let me know =:^)
>
>
> ---------------------------------------------------------------------
> This email is protected by LBackup, an open source backup solution.
> Free as in freedom; LBackup is licensed under the GNU GPL
> http://www.lbackup.org
>
>
>

> the -i shows that most are being copied due to time differences, so in
> theory -t should work? This does in fact work on a little test setup
> on my work laptop, i will test it properly when i get home tonight.


I am glad that the -t option is working with your test.

If you decide to use LBackup and have a full hard linked backup then a possible
option is to have a post action script which passes a list of files and
directories which have been updated or created to tar for archive creation.

To enable rsync session logging add the following line to the LBackup backup
configuration file :
 
 enable_rsync_session_log="YES"

Then using tar with the -n option you would be able to feed in a list of newly
created or modified files.

Depending upon what kind of information you want to preserve something like the
following could do the trick. However, please keep in mind that you may need to
add or remove tar various options.

 tar -c -n -z -v -f /tmp/new_archive.tar.gz -T /tmp/list_to_archive.txt 

If you opted for such an approach you could potentially stop using the
--compare-dest option and rely upon the output from rsync for a list of files
you would like to archive.

All the best with your backup script. 

With any luck the rsync -t option will work with you backup as well as it did
for your testing.


---------------------------
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org