rsync - Jul 2002 - Pipelining arbitrary file transfers + --{in,ex}clude oddness

I'd like to rsync a series of files at arbitrary points in a tree in one
connection and am struggling to figure out how to do this. rsync seems
AFAICS to have a fundamental src_file dst_file model, optionally with
recursion.

The best I could come up with was recursing plus multiple --include with
a final --exclude "*". The idea is to avoid multiple calls to rsync
and
thus TCP/session overhead costs. Is this reasonable? A better way?
(Sneaking feeling I'm missing something really obvious at 2am, despite
looking in the FAQ, man page, google and the wishlist...)

Anyway, doesn't this scheme make the server work hard building a
file list at its end?

**

In the course of playing with this I found something I can't explain.
On one server, this scheme seems to work fine but another says there's
nothing there. If I try to get the file directly it's reported there. In
other words, (missing final --exclude "*" for clarity)

	rsync -a rsync://server/module --include path/file # "nothing to do"
	rsync -a rsync://server/module/path/file           # shows file & path info

Here's a transcript to demonstrate this: (yes, I'm playing with apt-get
update speed-ups but perhaps ignore that for now :-)
Using: rsync  version 2.5.6cvs  protocol version 26

1. non-us seems to work fine:

$ rsync -Rzav --progress
rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages
receiving file list ... 
6 files to consider
drwxrwsr-x        4096 2002/07/12 01:26:18 dists
lrwxrwxrwx           3 2001/02/07 13:59:15 dists/testing -> woody
drwxrwsr-x        4096 2002/07/12 19:53:44 dists/testing/non-US
drwxrwsr-x        4096 2001/01/17 23:52:49 dists/testing/non-US/main
drwxrwsr-x        4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386
-rw-rw-r--      127844 2002/07/11 19:52:45
dists/testing/non-US/main/binary-i386/Packages
wrote 133 bytes  read 200 bytes  666.00 bytes/sec
total size is 127847  speedup is 383.92

2. security ditto:

$ rsync -Rzav --progress
rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages
receiving file list ... 
6 files to consider
drwxrwsr-x        4096 2002/05/01 13:14:24 dists
lrwxrwxrwx           5 2002/05/10 02:50:17 dists/testing -> woody
drwxrwsr-x        4096 2002/07/12 21:40:06 dists/testing/updates
drwxrwsr-x        4096 2002/05/01 13:14:13 dists/testing/updates/main
drwxrwsr-x        4096 2002/07/12 21:40:00
dists/testing/updates/main/binary-i386
-rw-rw-r--       88472 2002/07/12 21:22:39
dists/testing/updates/main/binary-i386/Packages
wrote 137 bytes  read 199 bytes  224.00 bytes/sec
total size is 88477  speedup is 263.32

3. back to non-us using the --include path/file --exclude "*" hack.
   All's well:

$ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include
dists/testing/non-US/main/binary-i386/Packages --exclude "*"
receiving file list ... 
1 file to consider
-rw-rw-r--      127844 2002/07/11 19:52:45
dists/testing/non-US/main/binary-i386/Packages
wrote 144 bytes  read 118 bytes  524.00 bytes/sec
total size is 127844  speedup is 487.95
asterix:/home/paulm#

4. this fails on security however:

$ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include
dists/testing/updates/main/binary-i386/Packages --exclude "*"
receiving file list ... 
0 files to consider
client: nothing to do: perhaps you need to specify some filenames or the
--recursive option?

Case 4 seems to apply for http.us.debian.org too. I've only seen non-us
work.

Cheers,
Paul

-- 
Paul Makepeace ....................................... http://paulm.com/

"If I thought it would help, then you can place a hammer in a cake."
   -- http://paulm.com/toys/surrealism/

Yes, doing explicit includes followed by exclude '*' is what people do.
Because you exclude '*' and the rsync algorithm is recursive, you need
to
explicitly include all parent directories of the files you want to copy
as well.

We have discussed making a --files-from option with which people can
explicitly list only the files they want to copy, and I had promised to
write it, but I haven't been able to find the time.

- Dave Dykstra


On Sat, Jul 13, 2002 at 02:43:12AM +0100, Paul Makepeace
wrote:
> I'd like to rsync a series of files at arbitrary points in a tree in
one
> connection and am struggling to figure out how to do this. rsync seems
> AFAICS to have a fundamental src_file dst_file model, optionally with
> recursion.
> 
> The best I could come up with was recursing plus multiple --include with
> a final --exclude "*". The idea is to avoid multiple calls to
rsync and
> thus TCP/session overhead costs. Is this reasonable? A better way?
> (Sneaking feeling I'm missing something really obvious at 2am, despite
> looking in the FAQ, man page, google and the wishlist...)
> 
> Anyway, doesn't this scheme make the server work hard building a
> file list at its end?
> 
> **
> 
> In the course of playing with this I found something I can't explain.
> On one server, this scheme seems to work fine but another says there's
> nothing there. If I try to get the file directly it's reported there.
In
> other words, (missing final --exclude "*" for clarity)
> 
> 	rsync -a rsync://server/module --include path/file # "nothing to
do"
> 	rsync -a rsync://server/module/path/file           # shows file & path
info
> 
> Here's a transcript to demonstrate this: (yes, I'm playing with
apt-get
> update speed-ups but perhaps ignore that for now :-)
> Using: rsync  version 2.5.6cvs  protocol version 26
> 
> 1. non-us seems to work fine:
> 
> $ rsync -Rzav --progress
rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages
> receiving file list ... 
> 6 files to consider
> drwxrwsr-x        4096 2002/07/12 01:26:18 dists
> lrwxrwxrwx           3 2001/02/07 13:59:15 dists/testing -> woody
> drwxrwsr-x        4096 2002/07/12 19:53:44 dists/testing/non-US
> drwxrwsr-x        4096 2001/01/17 23:52:49 dists/testing/non-US/main
> drwxrwsr-x        4096 2002/07/12 19:52:41
dists/testing/non-US/main/binary-i386
> -rw-rw-r--      127844 2002/07/11 19:52:45
dists/testing/non-US/main/binary-i386/Packages
> wrote 133 bytes  read 200 bytes  666.00 bytes/sec
> total size is 127847  speedup is 383.92
> 
> 2. security ditto:
> 
> $ rsync -Rzav --progress
rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages
> receiving file list ... 
> 6 files to consider
> drwxrwsr-x        4096 2002/05/01 13:14:24 dists
> lrwxrwxrwx           5 2002/05/10 02:50:17 dists/testing -> woody
> drwxrwsr-x        4096 2002/07/12 21:40:06 dists/testing/updates
> drwxrwsr-x        4096 2002/05/01 13:14:13 dists/testing/updates/main
> drwxrwsr-x        4096 2002/07/12 21:40:00
dists/testing/updates/main/binary-i386
> -rw-rw-r--       88472 2002/07/12 21:22:39
dists/testing/updates/main/binary-i386/Packages
> wrote 137 bytes  read 199 bytes  224.00 bytes/sec
> total size is 88477  speedup is 263.32
> 
> 3. back to non-us using the --include path/file --exclude "*"
hack.
>    All's well:
> 
> $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include
dists/testing/non-US/main/binary-i386/Packages --exclude "*"
> receiving file list ... 
> 1 file to consider
> -rw-rw-r--      127844 2002/07/11 19:52:45
dists/testing/non-US/main/binary-i386/Packages
> wrote 144 bytes  read 118 bytes  524.00 bytes/sec
> total size is 127844  speedup is 487.95
> asterix:/home/paulm#
> 
> 4. this fails on security however:
> 
> $ rsync -Rzav --progress rsync://security.debian.org/debian-security/
--include dists/testing/updates/main/binary-i386/Packages --exclude
"*"
> receiving file list ... 
> 0 files to consider
> client: nothing to do: perhaps you need to specify some filenames or the
--recursive option?
> 
> Case 4 seems to apply for http.us.debian.org too. I've only seen non-us
> work.
> 
> Cheers,
> Paul
> 
> -- 
> Paul Makepeace ....................................... http://paulm.com/
> 
> "If I thought it would help, then you can place a hammer in a
cake."
>    -- http://paulm.com/toys/surrealism/
> 
> -- 
> To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html