rsync - Feb 2005 - help troubleshooting inconsistencies in back up sizes

Hello list,

I'll first describe my set up:

server1 : live server
server2 : backup
server3 : backup of the backup

so the data set is copied in this order
server1->server2->server3

they are not done at the same time so there would be no collisions.

I use this shell script to back up:

for i in a b c d e f g h i j k l m n o p q r s t u v w x y z `seq 0 9`; do
        /usr/local/bin/rsync -a -z -W --delete /mailhome/$i/
user@backup:/mailhome/$i
done

question one would be does this look correct?

now here is the heart of the problem:
on server1 the partition is 121G
on server2 it's 110G
on server3 it's 118G

so I assume I have multiple problems here. I don't see --progress as being
usable in my case since I have such a large amount of files, how can I
debug what the differences are between these 3 body of files that doesn't
involve actually checking them individually? I basically want to be
informed of errors of any kind (io, permissions) as my logging of
stderr/stdout doesn't show anything, which makes me think many the problem
is my command line but it sure looks ok to me. I'm using version 2.6.3 of
rsync btw.

Thank you.

--
Never be afraid to tell the world who you are.
             -- Anonymous
 10:45:01 up 3 days, 21:08,  5 users,  load average: 0.17, 0.08, 0.06

On Thu, 3 Feb 2005, gaw zay <gauze@dropdead.org> wrote:
....
> I use this shell script to back up:
> 
> for i in a b c d e f g h i j k l m n o p q r s t u v w x y z `seq 0 9`; do
>         /usr/local/bin/rsync -a -z -W --delete /mailhome/$i/
user@backup:/mailhome/$i
> done
>
> question one would be does this look correct?
Your script looks OK, but it helps to use at least one -v option
so you can see what files are being transferred.  If you have a busy
mailserver at the time of the rsync, you might be needlessly
transferring lock files and temporary files as mailboxes are updated.

--stats would help you see the size of the hierarchies, too.

You can remove -v and --stats once you get an idea of the activity.
> now here is the heart of the problem:
> on server1 the partition is 121G
> on server2 it's 110G
> on server3 it's 118G
When you say the partition is NNN, do you mean that's the used space
as reported by df?
> so I assume I have multiple problems here. I don't see --progress as
being
> usable in my case since I have such a large amount of files, how can I
> debug what the differences are between these 3 body of files that
doesn't
> involve actually checking them individually? I basically want to be
> informed of errors of any kind (io, permissions) as my logging of
> stderr/stdout doesn't show anything, which makes me think many the
problem
> is my command line but it sure looks ok to me. I'm using version 2.6.3
of
> rsync btw.
If you don't have -v or --stats or --progress then there will be
no output for a successful rsync.

You should be getting any error messages that may be generated.

You could check the exit code from rsync and print your own message
if it is non-zero to ensure that you get a message when there's
an error.

To examine the difference between hierarchies, do this on each server
(replacing N with a number):

  find /mailhome -ls | sort +10 >/tmp/serverN.txt

(The sort sorts on the filename at the end of the line.)

If the machines use different timezones, put TZ=GMT before the find:

  TZ=GMT find /mailhome -ls | sort +10 >/tmp/serverN.txt

so all the timestamps will be the same.

Then use rsync to gather them on one machine and do diffs between them.
-- 
        John Van Essen  Univ of Minn. Alumnus  <vanes002@umn.edu>