rsync - Jan 2008 - preserving Mac OS X metadata in rsync backups and restores

Hello all,

I'm new to the list, but have done quite a bit of researching before  
regarding the support of Mac OS X specific features (resource forks,  
extended attributes, ACLs, file creation & modification date).

By reading the archives, I get the impression that the current  
version of rsync 3.0.0pre8 is quite far in this respect. At least it  
sounds so, and I thank the developers very much for this! I like your  
approach much more than the (very buggy) one originally pursued by  
Apple (store metadata in separate ._ file).

I plan to do the following:

* Run a Linux server (Ubuntu, I guess, on a ext3 partition) with two  
separate internal ATA hard disks formatted in XFS and configured in  
software RAID to store the actual backup data. (As I understand, I  
should use XFS rather than ext3 because XFS supports extended  
attributes large enough to hold also larger converted Mac resource  
forks.)

* Back up from different Mac OS X clients (cuurently all on 10.4, but  
I might upgrade them to 10.5 later) to the server using rsync over  
ssh. This should hopefully preserve (most of) the Mac-specifif  
metadata on the server. (Actually I plan to use rsnapshot, but I  
believe if I have rsync installed in the newest version and possibly  
tell rsnaphot to use the appropriate rsync options, things will be  
the same.)

Now my question is the following:

1) What would I have to do to ensure the metadata is also restored  
correctly? I assume I will have to use rsync for restoring also, and  
if I just copy over data (using, e.g, scp or over an AFP or CIFS or  
NFS network mount), I will lose this metadata. Is this correct?

Another problem I'm thinking about is that rsnapshot should be run on  
the server to "pull" the backups over the network. One cannot run it  
on the clients and "push" the data to the server over the network --  
which is what I'd prefer because I plan to not leave the server on  
all day but rather have it woken up by the (laptop) clients when  
needed who'll take care of the scheduling of the backups (using  
anachron or launchd etc.) One could, however, run rsnapshot on the  
clients to backup onto a locally attached storage device.

This leads me to the second question:

2) If I mount the server as a network drive on the clients using AFP,  
SMB/CIFS, NFS, ..., and then backup to this 'locally attached' drive  
with rsync (using rsnapshot), will I lose the metadata because of the  
transfer via the SMB/... layer?

Thanks a lot for a great program!
-Moritz

On Jan 18, 2008, at 8:14 AM, Moritz Heckscher wrote:
> Hello all,
>
> I'm new to the list, but have done quite a bit of researching before  
> regarding the support of Mac OS X specific features (resource forks,  
> extended attributes, ACLs, file creation & modification date).
>
> By reading the archives, I get the impression that the current  
> version of rsync 3.0.0pre8 is quite far in this respect. At least it  
> sounds so, and I thank the developers very much for this! I like  
> your approach much more than the (very buggy) one originally pursued  
> by Apple (store metadata in separate ._ file).
>
Be careful and test, test, test. I tried using pre8 to sync two local  
Xserve RAID's(about 2TB of data) and I'm seeing these errors.

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken  
pipe (32)
[receiver] internal abbrev error!
rsync error: error in rsync protocol data stream (code 12) at  
xattrs.c(565) [receiver=3.0.0pre8]
rsync: connection unexpectedly closed (175959 bytes received so far)  
[sender]
rsync error: error in rsync protocol data stream (code 12) at  
io.c(600) [sender=3.0.0pre8]

I have another Xserve RAID(about 1.3TB) and I don't get those errors  
when syncing with pre8. I'm trying to pin down what files/folders are  
causing the problem now.
> I plan to do the following:
>
> * Run a Linux server (Ubuntu, I guess, on a ext3 partition) with two  
> separate internal ATA hard disks formatted in XFS and configured in  
> software RAID to store the actual backup data. (As I understand, I  
> should use XFS rather than ext3 because XFS supports extended  
> attributes large enough to hold also larger converted Mac resource  
> forks.)
>
> * Back up from different Mac OS X clients (cuurently all on 10.4,  
> but I might upgrade them to 10.5 later) to the server using rsync  
> over ssh. This should hopefully preserve (most of) the Mac-specifif  
> metadata on the server. (Actually I plan to use rsnapshot, but I  
> believe if I have rsync installed in the newest version and possibly  
> tell rsnaphot to use the appropriate rsync options, things will be  
> the same.)
>
> Now my question is the following:
>
> 1) What would I have to do to ensure the metadata is also restored  
> correctly? I assume I will have to use rsync for restoring also, and  
> if I just copy over data (using, e.g, scp or over an AFP or CIFS or  
> NFS network mount), I will lose this metadata. Is this correct?
>
Why not use rsync3 for both backup and restore. Either use ssh (rsync - 
azXA --delete /path/to/source server:/path/to/target) or setup an  
rsync daemon server. This way you let rsync handle the metadata.


> Another problem I'm thinking about is that rsnapshot should be run  
> on the server to "pull" the backups over the network. One cannot
run
> it on the clients and "push" the data to the server over the
network
> -- which is what I'd prefer because I plan to not leave the server  
> on all day but rather have it woken up by the (laptop) clients when  
> needed who'll take care of the scheduling of the backups (using  
> anachron or launchd etc.) One could, however, run rsnapshot on the  
> clients to backup onto a locally attached storage device.
>
You don't need rsnapshot. Use the --link-dist option to create  
incremental backups.
> This leads me to the second question:
>
> 2) If I mount the server as a network drive on the clients using  
> AFP, SMB/CIFS, NFS, ..., and then backup to this 'locally attached'
> drive with rsync (using rsnapshot), will I lose the metadata because  
> of the transfer via the SMB/... layer?
>
> Thanks a lot for a great program!
> -Moritz
>
> -- 
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

David Miller.

On Fri, 2008-01-18 at 17:14 +0100, Moritz Heckscher
wrote:
> Another problem I'm thinking about is that rsnapshot should be run on  
> the server to "pull" the backups over the network. One cannot run
it
> on the clients and "push" the data to the server over the network
--
> which is what I'd prefer because I plan to not leave the server on  
> all day but rather have it woken up by the (laptop) clients when  
> needed who'll take care of the scheduling of the backups (using  
> anachron or launchd etc.) One could, however, run rsnapshot on the  
> clients to backup onto a locally attached storage device.
I would like to dispel once and for all the myth that rsnapshot cannot
be used with a push.  While it is /designed/ to pull backups, with a few
extra steps one can push a backup to an rsnapshot installation on
another machine.  I outlined the setup here:

http://lists.samba.org/archive/rsync/2007-December/019470.html

How are the clients going to wake the server?  Does the server wake up
on any incoming network connection (I hope)?
> 2) If I mount the server as a network drive on the clients using AFP,  
> SMB/CIFS, NFS, ..., and then backup to this 'locally attached'
drive
> with rsync (using rsnapshot), will I lose the metadata because of the  
> transfer via the SMB/... layer?
I would strongly discourage you from pushing the backup via a network
drive.  In addition to potential metadata mangling, you lose the benefit
of rsync's delta-transfer algorithm.

Matt