foner-rsync@media.mit.edu
2009-Jan-12 04:39 UTC
Is -R --link-dest really hard to use, or is it me?
I've got a problem for which the combination of -R and --link-dest doesn't seem to be quite enough---and I may have discovered a few small bugs as well; test cases are below. [And if someone has a scheme for doing this that doesn't involve rsync at all, but works okay, I'm all ears as well---I'm not the first with this problem.] Here's my problem: I unfortunately need to move a large dirvish vault. This is a directory tree consisting of -many- hardlinked files, which means that moving it in pieces will copy many times more data than is actually there, but trying to move the entire thing in one shot consumes more RAM than is available. [rsync on the toplevel dir blew up almost immediately, as I expected. cp -a was consuming at least 130meg per snapshot and therefore looked likely to consume at least 10G of RAM to finish; it's actually possible for other reasons it might have been closer to 20G. It thus got slower and slower as it became more and more page-bound and I eventually got tired of it thrashing itself to death; ETA might have been a few weeks at that rate. I can't just move the underlying blocks (e.g., copy the partition as a partition) because the whole reason I'm moving this filesystem in the first place is because it has errors that fsck is having trouble fixing---bug or bad hardware isn't established yet. And I don't know if dump/restore works well on ext3 filesystems, is well-tested these days, will work for ext4 when I finally migrate to that, or produces good data if the filesystem I'm starting with has errors that fsck complains about (or if it, too, will consume enormous amounts of RAM, but I'm assuming it's not trying to cache every inode it dumps, so maybe that might work if I trusted it---opinions anyone?)] So---rsync to the rescue, except not. A normal dirvish backup just uses --link-dest against the previous host/date combo, and works fine. I could copy the entire set of snapshots to a new filesystem the same way, EXCEPT for a problem: I took pains to hardlink files -across- hosts' backups that were also the same, so I didn't have a zillion copies of the same files that are all shared by most releases and any linux anyway. E.g., in this sort of arrangement: hostA/20080101 hostB/20080101 ... hostF/20080101 ... hostA/20080102 hostB/20080102 ... hostF/20080102 ... dirvish (well, rsync) itself hardlinked files between hostA/20080101 and hostA/20080102 on successive runs, and then -I- ran a tool (faster-dupemerge) that hardlinked identical files between hostA/20080101 and hostB/20080101 (etc). Once this is done across the very first set of dumps (e.g., 20080101 in this example), then even though rsync is doing --link-dest only from hostA to hostA on successive runs, everything stays hardlinked together across hosts because the same inode is being reused everywhere. (I also run faster-dupemerge across all hosts for the most-recent pair of backups to catch files that have been -copied or moved-, either from one dir to another on the same host, or across hosts. Works great.) Unfortunately, I can't get rsync to do the right thing when I'm trying to copy this structure. What I'd -like- to do is to take all of hostA..hostF---for a single date---and copy them all at once, using --link-dest to point back at the previous date's set of hosts as the basis. But because of the way the directories are structured, I need to use -R so I get the same structure recreated, and that seems to break --link-dest, unless there's some syntax issue in what I'm doing. Small test case: Imagine that "src" is my original filesystem, and "dst" is where I'm trying to move things. (Here, they share a superior directory, but of course in real life they're different filesystems.) "foo" is my test file; there are multiple copies of it in src that are all hardlinked together. I've already done the push of the first vault's contents from src to dst, so --link-dest has something to work with; note that the inode numbers for foo in src and dst are different (since, again, in real life, they're on different filesystems), but that all copies of foo in either src or dst (so far) share the same inode. The A, B, and C directories correspond to individual hosts. 18:45:42 ~/H$ find . -name "foo" -ls 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo 84426 4 -rw-r--r-- 1 blah blah 4 Jan 11 18:43 ./dst/a/1/foo 18:45:46 ~/H$ ~/rsync-3.0.5/rsync -aviH --link-dest=../1 src/a/2/ dst/a/2/ sending incremental file list created directory dst/a/2 cd..t...... ./ sent 61 bytes received 15 bytes 152.00 bytes/sec total size is 4 speedup is 0.05 18:46:11 ~/H$ find . -name "foo" -ls 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/1/foo 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/2/foo Okay, so the above shows that --link-dest without -R appears to work, BUT--- how come there was no actual output from rsync when it created dst/a/2/foo? Correct side-effect (foo created, with correct inode), but incorrect output. So the story thus far: 18:46:16 ~/H$ find . -ls 84408 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:44 . 84410 4 drwxr-xr-x 5 blah blah 4096 Jan 11 18:42 ./src 84411 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/a 84412 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:43 ./src/a/1 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo 84417 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:45 ./src/a/2 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo 84413 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/b 84414 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/b/1 84418 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/b/2 84415 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/c 84416 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/c/1 84419 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/c/2 84421 4 drwxr-xr-x 5 blah blah 4096 Jan 11 18:44 ./dst 84422 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:46 ./dst/a 84425 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:43 ./dst/a/1 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/1/foo 84427 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:45 ./dst/a/2 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/2/foo 84423 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:44 ./dst/b 84424 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:44 ./dst/c Created some more hardlinks to get the ball rolling on B & C. 19:02:46 ~/H/src$ ln ../dst/a/1/foo ../dst/b/1/foo 19:03:06 ~/H/src$ ln ../dst/a/1/foo ../dst/c/1/foo 19:03:10 ~/H/src$ find .. -name "foo" -ls 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/a/1/foo 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/a/2/foo 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/b/1/foo 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/c/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo 19:04:06 ~/H/src$ ln b/1/foo b/2/foo 19:04:26 ~/H/src$ ln c/1/foo c/2/foo 19:04:29 ~/H/src$ find .. -name "foo" -ls 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo 19:05:28 ~/H/src$ ~/rsync-3.0.5/rsync -n -aviH --link-dest=/home/blah/H/dst/a/1 b/2/ c/2/ dst/ sending incremental file list .d..t...... ./ sent 82 bytes received 15 bytes 194.00 bytes/sec total size is 8 speedup is 0.08 (DRY RUN) Can't really tell here what's going on 'cause of -n and output issue, I think. But without -R, probably wrong thing anyway. 19:05:42 ~/H/src$ ~/rsync-3.0.5/rsync -n -R -aviH --link-dest=/home/blah/H/dst/a/1 b/2/ c/2/ dst/ sending incremental file list cd+++++++++ b/ cd+++++++++ b/2/ cd+++++++++ c/ cd+++++++++ c/2/>f+++++++++ c/2/foohf+++++++++ b/2/foo => c/2/foo sent 173 bytes received 46 bytes 438.00 bytes/sec total size is 8 speedup is 0.04 (DRY RUN) Added -R. Hm. Note that it created a new c/2/foo and then hardlinked b/2/foo to it. Why didn't it hardlink c/2/foo to c/1/foo? Well, proably 'cause my link-dest is bogus---I'm trying to say, "The parent of the dirs I'm specifying" but I think that's getting tangled up because b/2 and c/2 are relative to src, --link-dest is relative to dst (but I'm forcing a rooted path 'cause "relative to dst" is just too confusing here), but neither b nor c is under a. But just specifying "...H/dst" isn't right, either, 'cause --link-dest doesn't match. (Tried it, didn't work.) So let's use multiple --link-dest's: 21:16:08 ~/H/src$ ~/rsync-3.0.5/rsync -n -R -aviH --link-dest=/home/blah/H/dst/b/1 --link-dest=/home/blah/H/dst/c/1 b/2/ c/2/ ../dst/ sending incremental file list .d..t...... b/2/ .d..t...... c/2/>f+++++++++ c/2/foohf+++++++++ b/2/foo => c/2/foo sent 167 bytes received 40 bytes 414.00 bytes/sec total size is 8 speedup is 0.04 (DRY RUN) Still no dice. 21:17:14 ~/H/src$ ~/rsync-3.0.5/rsync -R -aviH --link-dest=/home/blah/H/dst/b/1 --link-dest=/home/blah/H/dst/c/1 b/2/ c/2/ ../dst/ sending incremental file list .d..t...... b/2/ .d..t...... c/2/>f+++++++++ c/2/foohf+++++++++ b/2/foo => c/2/foo sent 211 bytes received 56 bytes 534.00 bytes/sec total size is 8 speedup is 0.03 21:17:45 ~/H/src$ find .. -name "foo" -ls 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo 84434 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ../dst/b/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo 84434 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ../dst/c/2/foo Yup, wrong inodes on the copy. 21:17:48 ~/H/src$ rm ../dst/b/2/foo ../dst/c/2/foo 21:18:04 ~/H/src$ find .. -name "foo" -ls 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/2/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/1/foo 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo Dropping -R of course puts the output in the wrong place (and what happened to "c/2", anyway?): 21:18:05 ~/H/src$ ~/rsync-3.0.5/rsync -n -aviH --link-dest=/home/blah/H/dst/b/1 --link-dest=/home/blah/H/dst/c/1 b/2 c/2 ../dst/ sending incremental file list cd+++++++++ 2/>f+++++++++ 2/foosent 89 bytes received 19 bytes 216.00 bytes/sec total size is 8 speedup is 0.07 (DRY RUN) I've tried adding slashes at various ends and other permutations, but nothing works. I can't seem to get -R and --link-dest to play nice. [Oh, and, btw, using ~ in --link-dest seems to confuse it; I had to drop back to /home/blah instead. Eh?] Operationally, this means that I have to copy every single host -and- date -separately-, hence multiplying the number of directory scans by the number of hosts -and- breaking all those carefully-created hardlinks among the hosts. I -think- that (as when I created the vault) hardlinking everybody in the very first vault and then going forward one host at a time, one data at a time, might work, BUT it misses movement/copy across the hosts (though I'm not sure if rsync could do this anyway--- I haven't worked up a test case yet; shouldn't it just do the right thing if it's -H mode and a file moves across directories but it still knows everyone's inode? I'm not sure.) I -think- I might be able to finesse this by actually physically rearranging the directories on the source---risky given that fsck is complaiing about it, but maybe... the idea would be to invert the organization so that every host is under a date (e.g., instead of hostA/date1, hostA/date2, etc, I make it date1/hostA, date1/hostB), and then I can specify a SINGLE dir (namely "date1") and not use -R. [I can't just specify a single HOST in the current arrangement because there are far more dates than hsots and that causes a huge directory scan that runs rsync out of memory.] The other tragic thing about this whole shebang is that, even though I'm doing successive runs pivoting around a date (e.g., each run uses the immediately-preceeding dir as its link-dest), I can't save the directory scan involved---even though the rsync that's quitting knows -exactly- what the rsync that's starting wants to know about that tree. If only there was a way to share state between them... (No, it doesn't appear to fit in the Linux filesystem cache, alas.)
I regret the slow response. I was interested in your problem, but I knew it would take me a while to respond thoughtfully, so I put the message aside and didn't get back to it until now. I hope this is still useful. On Sun, 2009-01-11 at 23:24 -0500, foner-rsync@media.mit.edu wrote:> I've got a problem for which the combination of -R and --link-dest > doesn't seem to be quite enough---and I may have discovered a few > small bugs as well; test cases are below. > > [And if someone has a scheme for doing this that doesn't involve rsync > at all, but works okay, I'm all ears as well---I'm not the first with > this problem.] > > Here's my problem: I unfortunately need to move a large dirvish > vault. This is a directory tree consisting of -many- hardlinked > files, which means that moving it in pieces will copy many times more > data than is actually there, but trying to move the entire thing in > one shot consumes more RAM than is available. [rsync on the toplevel > dir blew up almost immediately, as I expected. cp -a was consuming at > least 130meg per snapshot and therefore looked likely to consume at > least 10G of RAM to finish; it's actually possible for other reasons > it might have been closer to 20G. It thus got slower and slower as > it became more and more page-bound and I eventually got tired of it > thrashing itself to death; ETA might have been a few weeks at that > rate. I can't just move the underlying blocks (e.g., copy the > partition as a partition) because the whole reason I'm moving this > filesystem in the first place is because it has errors that fsck is > having trouble fixing---bug or bad hardware isn't established yet. > And I don't know if dump/restore works well on ext3 filesystems, is > well-tested these days, will work for ext4 when I finally migrate to > that, or produces good data if the filesystem I'm starting with has > errors that fsck complains about (or if it, too, will consume enormous > amounts of RAM, but I'm assuming it's not trying to cache every inode > it dumps, so maybe that might work if I trusted it---opinions > anyone?)] > > So---rsync to the rescue, except not. A normal dirvish backup just > uses --link-dest against the previous host/date combo, and works fine. > I could copy the entire set of snapshots to a new filesystem the same > way, EXCEPT for a problem: I took pains to hardlink files -across- > hosts' backups that were also the same, so I didn't have a zillion > copies of the same files that are all shared by most releases and any > linux anyway. E.g., in this sort of arrangement: > hostA/20080101 > hostB/20080101 > ... > hostF/20080101 > ... > hostA/20080102 > hostB/20080102 > ... > hostF/20080102 > ... > > dirvish (well, rsync) itself hardlinked files between hostA/20080101 > and hostA/20080102 on successive runs, and then -I- ran a tool > (faster-dupemerge) that hardlinked identical files between > hostA/20080101 and hostB/20080101 (etc). Once this is done across the > very first set of dumps (e.g., 20080101 in this example), then even > though rsync is doing --link-dest only from hostA to hostA on > successive runs, everything stays hardlinked together across hosts > because the same inode is being reused everywhere. (I also run > faster-dupemerge across all hosts for the most-recent pair of backups > to catch files that have been -copied or moved-, either from one dir > to another on the same host, or across hosts. Works great.) > > Unfortunately, I can't get rsync to do the right thing when I'm trying > to copy this structure. What I'd -like- to do is to take all of > hostA..hostF---for a single date---and copy them all at once, using > --link-dest to point back at the previous date's set of hosts as the > basis. But because of the way the directories are structured, I need > to use -R so I get the same structure recreated, and that seems to > break --link-dest, unless there's some syntax issue in what I'm doing.> Small test case: > > Imagine that "src" is my original filesystem, and "dst" is where I'm > trying to move things. (Here, they share a superior directory, but of > course in real life they're different filesystems.) "foo" is my test > file; there are multiple copies of it in src that are all hardlinked > together. I've already done the push of the first vault's contents > from src to dst, so --link-dest has something to work with; note that > the inode numbers for foo in src and dst are different (since, again, > in real life, they're on different filesystems), but that all copies > of foo in either src or dst (so far) share the same inode. The A, B, > and C directories correspond to individual hosts. > > 18:45:42 ~/H$ find . -name "foo" -ls > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo > 84426 4 -rw-r--r-- 1 blah blah 4 Jan 11 18:43 ./dst/a/1/foo > 18:45:46 ~/H$ ~/rsync-3.0.5/rsync -aviH --link-dest=../1 src/a/2/ dst/a/2/ > sending incremental file list > created directory dst/a/2 > cd..t...... ./ > > sent 61 bytes received 15 bytes 152.00 bytes/sec > total size is 4 speedup is 0.05 > 18:46:11 ~/H$ find . -name "foo" -ls > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo > 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/1/foo > 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/2/foo > > Okay, so the above shows that --link-dest without -R appears to work, BUT--- > how come there was no actual output from rsync when it created dst/a/2/foo? > Correct side-effect (foo created, with correct inode), but incorrect output.The lack of output here is by design. That's not to say that I think the design is a good one. IIUC, the current interpretation of a basis dir (--*-dest) is a source of files that "might as well be" in the corresponding places in the destination directory. When rsync copies a source file that has no counterpart in the destination and selects an "alternate basis" file from a --*-dest dir, it itemizes as if the alternate basis file were actually present in the destination. In your example, rsync considered the hard-linking from dst/a/1/foo to dst/a/2/foo to not be a change, so you would have to pass -ii to see it. This interpretation makes complete sense with --compare-dest, which was the first --*-dest option to be added, in 1998: http://gitweb.samba.org/?p=rsync.git;a=commit;h=375a4556c7a1ffb9a4e7117f33fc42ed2bc4c026 However, the more recently added --copy-dest and --link-dest: http://gitweb.samba.org/?p=rsync.git;a=commit;h=1de3e99bc5781a119c3c7a4aa176eb77a7039714 http://gitweb.samba.org/?p=rsync.git;a=commit;h=59c95e4243749273fe91f8197a39f89e4d905cb8 have the IMHO more useful interpretation that the basis dir is to be used as an optimization (of network traffic and/or destination disk usage), without affecting either the itemization or the final contents of the destination. I entered an enhancement request for this to be supported properly: https://bugzilla.samba.org/show_bug.cgi?id=5645> So the story thus far: > > 18:46:16 ~/H$ find . -ls > 84408 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:44 . > 84410 4 drwxr-xr-x 5 blah blah 4096 Jan 11 18:42 ./src > 84411 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/a > 84412 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:43 ./src/a/1 > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/1/foo > 84417 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:45 ./src/a/2 > 84420 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./src/a/2/foo > 84413 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/b > 84414 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/b/1 > 84418 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/b/2 > 84415 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:42 ./src/c > 84416 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/c/1 > 84419 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:42 ./src/c/2 > 84421 4 drwxr-xr-x 5 blah blah 4096 Jan 11 18:44 ./dst > 84422 4 drwxr-xr-x 4 blah blah 4096 Jan 11 18:46 ./dst/a > 84425 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:43 ./dst/a/1 > 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/1/foo > 84427 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:45 ./dst/a/2 > 84426 4 -rw-r--r-- 2 blah blah 4 Jan 11 18:43 ./dst/a/2/foo > 84423 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:44 ./dst/b > 84424 4 drwxr-xr-x 2 blah blah 4096 Jan 11 18:44 ./dst/c > > Created some more hardlinks to get the ball rolling on B & C. > > 19:02:46 ~/H/src$ ln ../dst/a/1/foo ../dst/b/1/foo > 19:03:06 ~/H/src$ ln ../dst/a/1/foo ../dst/c/1/foo > 19:03:10 ~/H/src$ find .. -name "foo" -ls > 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/a/1/foo > 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/a/2/foo > 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/b/1/foo > 84420 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../src/c/1/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo > > 19:04:06 ~/H/src$ ln b/1/foo b/2/foo > 19:04:26 ~/H/src$ ln c/1/foo c/2/foo > 19:04:29 ~/H/src$ find .. -name "foo" -ls > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/1/foo > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/a/2/foo > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/1/foo > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/b/2/foo > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/1/foo > 84420 4 -rw-r--r-- 6 blah blah 4 Jan 11 18:43 ../src/c/2/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/1/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/a/2/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/b/1/foo > 84426 4 -rw-r--r-- 4 blah blah 4 Jan 11 18:43 ../dst/c/1/foo > > 19:05:28 ~/H/src$ ~/rsync-3.0.5/rsync -n -aviH --link-dest=/home/blah/H/dst/a/1 b/2/ c/2/ dst/ > sending incremental file list > .d..t...... ./ > > sent 82 bytes received 15 bytes 194.00 bytes/sec > total size is 8 speedup is 0.08 (DRY RUN) > > Can't really tell here what's going on 'cause of -n and output issue, > I think. But without -R, probably wrong thing anyway.Without -R, the sources b/2/ and c/2/ are both placed at the root of the file list. The first source takes priority, so b/2/foo would be mapped to dst/foo by means of a hard link from /home/blah/H/dst/a/1/foo . As in the previous example, rsync doesn't consider this worthy of output.> 19:05:42 ~/H/src$ ~/rsync-3.0.5/rsync -n -R -aviH --link-dest=/home/blah/H/dst/a/1 b/2/ c/2/ dst/ > sending incremental file list > cd+++++++++ b/ > cd+++++++++ b/2/ > cd+++++++++ c/ > cd+++++++++ c/2/ > >f+++++++++ c/2/foo > hf+++++++++ b/2/foo => c/2/foo > > sent 173 bytes received 46 bytes 438.00 bytes/sec > total size is 8 speedup is 0.04 (DRY RUN) > > Added -R. Hm. Note that it created a new c/2/foo and then hardlinked > b/2/foo to it. Why didn't it hardlink c/2/foo to c/1/foo? Well, > proably 'cause my link-dest is bogus---I'm trying to say, "The parent > of the dirs I'm specifying" but I think that's getting tangled up > because b/2 and c/2 are relative to src, --link-dest is relative to > dst (but I'm forcing a rooted path 'cause "relative to dst" is just > too confusing here), but neither b nor c is under a. But just > specifying "...H/dst" isn't right, either, 'cause --link-dest > doesn't match. (Tried it, didn't work.)Right. To recap the problem: In order to transfer both b/2/ and c/2/ to the proper places under dst/ in a single run, you needed to include the "b/2/" and "c/2/" path information in the file list by using -R. But consequently, rsync is going to look for b/2/foo and c/2/foo under whatever --link-dest dir you specify, and there's no directory on the destination side that contains files at those paths (yet).> So let's use multiple --link-dest's: > > 21:16:08 ~/H/src$ ~/rsync-3.0.5/rsync -n -R -aviH --link-dest=/home/blah/H/dst/b/1 --link-dest=/home/blah/H/dst/c/1 b/2/ c/2/ ../dst/ > sending incremental file list > .d..t...... b/2/ > .d..t...... c/2/ > >f+++++++++ c/2/foo > hf+++++++++ b/2/foo => c/2/foo > > sent 167 bytes received 40 bytes 414.00 bytes/sec > total size is 8 speedup is 0.04 (DRY RUN) > > Still no dice.Same problem: rsync is looking for /home/blah/H/dst/b/1/b/2/foo . [Skipping to the next example...]> Dropping -R of course puts the output in the wrong place (and what > happened to "c/2", anyway?): > > 21:18:05 ~/H/src$ ~/rsync-3.0.5/rsync -n -aviH --link-dest=/home/blah/H/dst/b/1 --link-dest=/home/blah/H/dst/c/1 b/2 c/2 ../dst/ > sending incremental file list > cd+++++++++ 2/ > >f+++++++++ 2/foo > > sent 89 bytes received 19 bytes 216.00 bytes/sec > total size is 8 speedup is 0.07 (DRY RUN)Without -R, file-list path information starts at the last slash in the source argument. That's why you're getting "2/foo". If the sources were "b/2/" and "c/2/" with trailing slashes, you would get just "foo".> I've tried adding slashes at various ends and other permutations, but > nothing works. I can't seem to get -R and --link-dest to play nice. > > [Oh, and, btw, using ~ in --link-dest seems to confuse it; I had to > drop back to /home/blah instead. Eh?]Tilde expansion is the shell's job. The shell checks for a tilde only at the start of each argument, since you don't want it to expand tildes in the middle of a path like "my-file.txt~2008-01-25~" ; and it isn't smart enough to realize that the tilde in "--link-dest=~/test/something" *is* the start of a path. If you want tilde expansion, use the unsticked form of the option: "--link-dest ~/test/something".> Operationally, this means that I have to copy every single host -and- > date -separately-, hence multiplying the number of directory scans by > the number of hosts -and- breaking all those carefully-created hardlinks > among the hosts.I think using a separate rsync run for each hostX/DATE dir is the way to go since it's easy to specify an appropriate --link-dest dir, or more than one. With this approach, you don't need -H unless you want to preserve hard links among a single host's files on a single day. In recent months, several rsnapshot users have posted about migration problems similar to yours but one-dimensional (dates only), and I wrote a script called "rsnapshot-copy" to automate the process of copying the dates one by one, each time with --link-dest to the previous date: http://rsnapshot.cvs.sourceforge.net/viewvc/rsnapshot/rsnapshot/utils/rsnapshot-copy?view=markup You may wish to read the thread from which rsnapshot-copy originated for more insights: http://sourceforge.net/mailarchive/forum.php?thread_name=47FBD95C.2080906%40cfa.harvard.edu&forum_name=rsnapshot-discuss You could use it as a starting point for a more sophisticated script for your scenario. Just loop through every (host, date) pair and run rsync, passing as many --link-dest options as you need to help rsync discover all the inter-host and inter-date links. My inclination would be to make the dates the outer loop and the hosts the inner loop since you have only six hosts but presumably many more dates. Then, for each hostX/DATE dir, I would --link-dest to each host's most recent existing dir on the destination. E.g., if you go through the hosts in alphabetical order, hostC/20080102 would --link-dest to hostA/20080102 and hostB/20080102 (already copied for this date) as well as hostC/20080101, hostD/20080101, hostE/20080101, and hostF/20080101 (hosts not yet copied for 20080102). You could try this and adjust as you see fit. Rsync does a linear search through the basis dirs, so you should put the most likely ones first, e.g., hostC/20080101 in my example. In deciding how many dirs to pass, consider the benefit of the extra dirs versus the time that rsync wastes checking those dirs for files that are genuinely new in the current hostX/DATE dir.> I -think- I might be able to finesse this by actually physically > rearranging the directories on the source---risky given that fsck > is complaiing about it, but maybe... the idea would be to invert > the organization so that every host is under a date (e.g., instead > of hostA/date1, hostA/date2, etc, I make it date1/hostA, date1/hostB), > and then I can specify a SINGLE dir (namely "date1") and not use -R. > [I can't just specify a single HOST in the current arrangement because > there are far more dates than hsots and that causes a huge directory > scan that runs rsync out of memory.]That would work. To avoid physically rearranging the source, you could create a structure of symlinks on another filesystem and point rsync to that. The downside is that you have to use -H to catch cross-host hard links. -- Matt