Raphael Becker
2008-May-31 19:48 UTC
rsync 3.0.2 with --fileflags on FreeBSD: cannot rsync hardlinked immutable files
Hi *, it seems rsync with --fileflags isn't able to work on (already) hardlinked and immutable ("schg") files on FreeBSD. The following scripts will create a simple example for this behaviour: -------------------------------------------------------------- #! /bin/sh # # set -x DIR="/var/tmp/rsync_$(date +%s)/" mkdir "${DIR}/" # Preparing dir_A mkdir "${DIR}/dir_A/" touch "${DIR}/dir_A/file_A" ln "${DIR}/dir_A/file_A" "${DIR}/dir_A/file_B" chflags schg "${DIR}/dir_A/file_A" ls -laio "${DIR}/dir_A/" # Try rsync: will fail because of schg'ed hardlinked file /usr/local/bin/rsync -avHWx --fileflags "${DIR}/dir_A/" "${DIR}/dir_B/" ls -laio "${DIR}/dir_B/" # Try cpdup: will work /usr/local/bin/cpdup "${DIR}/dir_A/" "${DIR}/dir_C/" ls -laio "${DIR}/dir_C/" -------------------------------------------------------------- will output: -------------------------------------------------------------- + date +%s + DIR=/var/tmp/rsync_1212259951/ + mkdir /var/tmp/rsync_1212259951// + mkdir /var/tmp/rsync_1212259951//dir_A/ + touch /var/tmp/rsync_1212259951//dir_A/file_A + ln /var/tmp/rsync_1212259951//dir_A/file_A /var/tmp/rsync_1212259951//dir_A/file_B + chflags schg /var/tmp/rsync_1212259951//dir_A/file_A + ls -laio /var/tmp/rsync_1212259951//dir_A/ total 4 188674 drwxr-xr-x 2 root wheel - 512 May 31 20:52 . 188673 drwxr-xr-x 3 root wheel - 512 May 31 20:52 .. 188676 -rw-r--r-- 2 root wheel schg 0 May 31 20:52 file_A 188676 -rw-r--r-- 2 root wheel schg 0 May 31 20:52 file_B + /usr/local/bin/rsync -avHWx --fileflags /var/tmp/rsync_1212259951//dir_A/ /var/tmp/rsync_1212259951//dir_B/ sending incremental file list created directory /var/tmp/rsync_1212259951//dir_B ./ file_B rsync: link "/var/tmp/rsync_1212259951/dir_B/file_A" => file_B failed: Operation not permitted (1) sent 95 bytes received 34 bytes 258.00 bytes/sec total size is 0 speedup is 0.00 rsync error: some files could not be transferred (code 23) at main.c(1031) [sender=3.0.2] + ls -laio /var/tmp/rsync_1212259951//dir_B/ total 4 188677 drwxr-xr-x 2 root wheel - 512 May 31 20:52 . 188673 drwxr-xr-x 4 root wheel - 512 May 31 20:52 .. 188678 -rw-r--r-- 1 root wheel schg 0 May 31 20:52 file_B + /usr/local/bin/cpdup /var/tmp/rsync_1212259951//dir_A/ /var/tmp/rsync_1212259951//dir_C/ + ls -laio /var/tmp/rsync_1212259951//dir_C/ total 4 188679 drwxr-xr-x 2 root wheel - 512 May 31 20:52 . 188673 drwxr-xr-x 5 root wheel - 512 May 31 20:52 .. 188680 -rw-r--r-- 2 root wheel schg 0 May 31 20:52 file_A 188680 -rw-r--r-- 2 root wheel schg 0 May 31 20:52 file_B -------------------------------------------------------------- It seems rsync first sets the immutable-flag ("schg") and try to hardlink the second file on the (already) schg'ed file, which fails as expected. A workaround would be to set alle hardlinks before setting the immutable-flags on the files. I got a patched patch for "flags.diff" by Rolf Grossmann for rsync 2.6.9, which handled hardlinks on schg'ed files gracefully (using --flags instead of --fileflags), see http://rabe.uugrn.org/files/replacement_for_net_rsync_work_rsync-2.6.9_patches_flags.diff -------------------------------------------------------------- + date +%s + DIR=/var/tmp/rsync_1212261208/ + mkdir /var/tmp/rsync_1212261208// + mkdir /var/tmp/rsync_1212261208//dir_A/ + touch /var/tmp/rsync_1212261208//dir_A/file_A + ln /var/tmp/rsync_1212261208//dir_A/file_A /var/tmp/rsync_1212261208//dir_A/file_B + chflags schg /var/tmp/rsync_1212261208//dir_A/file_A + ls -laio /var/tmp/rsync_1212261208//dir_A/ total 4 212060 drwxr-xr-x 2 root wheel - 512 May 31 21:13 . 212059 drwxr-xr-x 3 root wheel - 512 May 31 21:13 .. 212061 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_A 212061 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_B + /usr/local/bin/rsync -avHWx --flags /var/tmp/rsync_1212261208//dir_A/ /var/tmp/rsync_1212261208//dir_B/ building file list ... done created directory /var/tmp/rsync_1212261208//dir_B ./ file_B file_A => file_B sent 148 bytes received 61 bytes 418.00 bytes/sec total size is 0 speedup is 0.00 + ls -laio /var/tmp/rsync_1212261208//dir_B/ total 4 212062 drwxr-xr-x 2 root wheel - 512 May 31 21:13 . 212059 drwxr-xr-x 4 root wheel - 512 May 31 21:13 .. 212063 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_A 212063 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_B + /usr/local/bin/cpdup /var/tmp/rsync_1212261208//dir_A/ /var/tmp/rsync_1212261208//dir_C/ + ls -laio /var/tmp/rsync_1212261208//dir_C/ total 4 212064 drwxr-xr-x 2 root wheel - 512 May 31 21:13 . 212059 drwxr-xr-x 5 root wheel - 512 May 31 21:13 .. 212065 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_A 212065 -rw-r--r-- 2 root wheel schg 0 May 31 21:13 file_B -------------------------------------------------------------- root@bottom:~# pkg_info | grep rsync rsync-2.6.9_2 A network file distribution/synchronization utility -------------------------------------------------------------- I use rsync for lot of things, especially rsyncing FreeBSD jails between jailhosts (via ssh) needs the hardlinks on schg'ed files, for local-to-local syncing cpdup is doing fine here. Regards Raphael Becker -- -- Raphael Becker <rabe@uugrn.org> http://rabe.uugrn.org/ GnuPG: E7B2 1D66 3AF2 EDC7 9828 6D7A 9CDA 3E7B 10CA 9F2D .........|.........|.........|.........|.........|.........|.........|.. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20080531/8254cf10/attachment.bin
Raphael Becker
2008-Jun-07 22:55 UTC
rsync 3.0.2 with --fileflags on FreeBSD: cannot rsync hardlinked immutable files
On Sat, May 31, 2008 at 09:16:43PM +0200, I wrote:> Hi *, > > it seems rsync with --fileflags isn't able to work on (already) > hardlinked and immutable ("schg") files on FreeBSD. The following > scripts will create a simple example for this behaviour:I forgot to check --force-change but this didn't help either. I googled around and found http://rsync.samba.org/ftp/rsync/patches/fileflags.diff ... which doesn't seem to work on the current(?) sources: root@daemon:/usr/ports/net/rsync# make extract ===> Vulnerability check disabled, database not found ===> Found saved configuration for rsync-3.0.2_1 => rsync-3.0.2.tar.gz doesn't seem to exist in /data/ftp/pub/FreeBSD/distfiles//. => Attempting to fetch from http://ftp2.de.freebsd.org/FreeBSD/distfiles//. fetch: http://ftp2.de.freebsd.org/FreeBSD/distfiles//rsync-3.0.2.tar.gz: Not Found => Attempting to fetch from http://rsync.samba.org/ftp/rsync/. rsync-3.0.2.tar.gz 100% of 747 kB 21 MBps => rsync-patches-3.0.2.tar.gz doesn't seem to exist in /data/ftp/pub/FreeBSD/distfiles//. => Attempting to fetch from http://ftp2.de.freebsd.org/FreeBSD/distfiles//. fetch: http://ftp2.de.freebsd.org/FreeBSD/distfiles//rsync-patches-3.0.2.tar.gz: Not Found => Attempting to fetch from http://rsync.samba.org/ftp/rsync/. rsync-patches-3.0.2.tar.gz 100% of 150 kB 17 MBps ===> Extracting for rsync-3.0.2_1 => MD5 Checksum OK for rsync-3.0.2.tar.gz. => SHA256 Checksum OK for rsync-3.0.2.tar.gz. => MD5 Checksum OK for rsync-patches-3.0.2.tar.gz. => SHA256 Checksum OK for rsync-patches-3.0.2.tar.gz. ===> rsync-3.0.2_1 depends on file: /usr/local/bin/perl5.8.8 - found Overriding the patch from the distfile with the patch from the web: root@daemon:/usr/ports/net/rsync# wget -O /data/tmp/usr/ports/net/rsync/work/rsync-3.0.2/patches/fileflags.diff http://rsync.samba.org/ftp/rsync/patches/fileflags.diff But this patch doesn't match the current(?) sources: root@daemon:/usr/ports/net/rsync# make patch ===> Patching for rsync-3.0.2_1 ===> rsync-3.0.2_1 depends on file: /usr/local/bin/perl5.8.8 - found ===> Applying extra patch /data/tmp/usr/ports/net/rsync/work/rsync-3.0.2/patches/fileflags.diff 1 out of 2 hunks failed--saving rejects to t_stub.c.rej 1 out of 4 hunks failed--saving rejects to util.c.rej *** Error code 2 Stop in /usr/ports/net/rsync. *** Error code 1 Stop in /usr/ports/net/rsync. The two distfiles are MD5 (rsync-3.0.2.tar.gz) = fd4c5d77d8cb7bb86ab209076fa214d9 MD5 (rsync-patches-3.0.2.tar.gz) = c076661c447ddc165f4ff427e7264a31 Does anyone know if there's a way to rsync hardlinks on immutable files? Wayne Davison? Rolf Grossmann? Anyone else? Help would be greatly appreciated. Regards Raphael Becker -- Raphael Becker <rabe@uugrn.org> http://rabe.uugrn.org/ GnuPG: E7B2 1D66 3AF2 EDC7 9828 6D7A 9CDA 3E7B 10CA 9F2D .........|.........|.........|.........|.........|.........|.........|.. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20080608/01a02f2f/attachment.bin