rsync - Mar 2003 - single rsync of large directory structure versus several smaller rsync's

Hello,

I am working on a backup script using rsync and have a question.

Rather than doing a single rsync of /home/ I have set my script up
to loop over /etc/passwd and do a seperate rsync of each user home dir 
(/home/user1, /home/user2) etc.

In the end, I should have the exact same backup either way. However I feel 
safest doing it as multiple rsyncs (one for each user). I am doing this 
because I want to avoid a case in which some fatal error on a particular dir 
kills off the whole rsync. For example, given:

/home/adoe
/home/jdoe
/home/zdoe

If I do:

rsync -vaze ssh /home/

I am afraid that if /home/adoe had a fatel error (one that kills rsync) all 
dirs past it would not be backed up and given an over-worked sysadmin 
(never!) we might miss the error until the day we need the backups.

If I do a seperate:

rsync -vaze ssh /home/adoe
rsync -vaze ssh /home/jdoe
rsync -vaze ssh /home/zdoe

etc.

I should still have a backup of each dir even if some fail.

Is this being too paranoid?

Is there something built into rsync to avoid this?

Does anyone have any better ideas for me? I want to make the best use of 
rsync possible.

Thanks for your advice and comments!



_________________________________________________________________
Add photos to your e-mail with MSN 8. Get 2 months FREE*.  
http://join.msn.com/?page=features/featuredemail

On Sun, Mar 16, 2003 at 09:05:45AM +0000, Pete Mocat
wrote:
> Hello,
> 
> I am working on a backup script using rsync and have a question.
> 
> Rather than doing a single rsync of /home/ I have set my script up
> to loop over /etc/passwd and do a seperate rsync of each user home dir 
> (/home/user1, /home/user2) etc.
> 
> In the end, I should have the exact same backup either way. However I feel 
> safest doing it as multiple rsyncs (one for each user). I am doing this 
> because I want to avoid a case in which some fatal error on a particular 
> dir kills off the whole rsync. For example, given:
> 
> /home/adoe
> /home/jdoe
> /home/zdoe
> 
> If I do:
> 
> rsync -vaze ssh /home/
> 
> I am afraid that if /home/adoe had a fatel error (one that kills rsync) all
> dirs past it would not be backed up and given an over-worked sysadmin 
> (never!) we might miss the error until the day we need the backups.
> 
> If I do a seperate:
> 
> rsync -vaze ssh /home/adoe
> rsync -vaze ssh /home/jdoe
> rsync -vaze ssh /home/zdoe
> 
> etc.
> 
> I should still have a backup of each dir even if some fail.
> 
> Is this being too paranoid?
How paranoid is too paranoid?
> Is there something built into rsync to avoid this?
If an error looks to be isolated to a single file rsync
continues onward and merely logs the error.  Your script
should examine the exit status and/or logs and notify if
there is an error.
> Does anyone have any better ideas for me? I want to make the best use of 
> rsync possible.
> 
> Thanks for your advice and comments!
I don't think you are being paranoid enough.  Unless your
backups are subsequently backed up again you only have
one backup here.  What is going to happen if a file gets
trashed and it isn't noticed for a few days?

I would suggest you look into Mike Rubel's snapshot
system, Dirvish or another system that has already dealt
with these issues and can maintain and manage multiple
backup images.  I cannot speak for the others but i know
that Dirvish is in production use at multiple sites.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt