Paul Makepeace
2002-Jul-12 18:51 UTC
Pipelining arbitrary file transfers + --{in,ex}clude oddness
I'd like to rsync a series of files at arbitrary points in a tree in one connection and am struggling to figure out how to do this. rsync seems AFAICS to have a fundamental src_file dst_file model, optionally with recursion. The best I could come up with was recursing plus multiple --include with a final --exclude "*". The idea is to avoid multiple calls to rsync and thus TCP/session overhead costs. Is this reasonable? A better way? (Sneaking feeling I'm missing something really obvious at 2am, despite looking in the FAQ, man page, google and the wishlist...) Anyway, doesn't this scheme make the server work hard building a file list at its end? ** In the course of playing with this I found something I can't explain. On one server, this scheme seems to work fine but another says there's nothing there. If I try to get the file directly it's reported there. In other words, (missing final --exclude "*" for clarity) rsync -a rsync://server/module --include path/file # "nothing to do" rsync -a rsync://server/module/path/file # shows file & path info Here's a transcript to demonstrate this: (yes, I'm playing with apt-get update speed-ups but perhaps ignore that for now :-) Using: rsync version 2.5.6cvs protocol version 26 1. non-us seems to work fine: $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages receiving file list ... 6 files to consider drwxrwsr-x 4096 2002/07/12 01:26:18 dists lrwxrwxrwx 3 2001/02/07 13:59:15 dists/testing -> woody drwxrwsr-x 4096 2002/07/12 19:53:44 dists/testing/non-US drwxrwsr-x 4096 2001/01/17 23:52:49 dists/testing/non-US/main drwxrwsr-x 4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386 -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages wrote 133 bytes read 200 bytes 666.00 bytes/sec total size is 127847 speedup is 383.92 2. security ditto: $ rsync -Rzav --progress rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages receiving file list ... 6 files to consider drwxrwsr-x 4096 2002/05/01 13:14:24 dists lrwxrwxrwx 5 2002/05/10 02:50:17 dists/testing -> woody drwxrwsr-x 4096 2002/07/12 21:40:06 dists/testing/updates drwxrwsr-x 4096 2002/05/01 13:14:13 dists/testing/updates/main drwxrwsr-x 4096 2002/07/12 21:40:00 dists/testing/updates/main/binary-i386 -rw-rw-r-- 88472 2002/07/12 21:22:39 dists/testing/updates/main/binary-i386/Packages wrote 137 bytes read 199 bytes 224.00 bytes/sec total size is 88477 speedup is 263.32 3. back to non-us using the --include path/file --exclude "*" hack. All's well: $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include dists/testing/non-US/main/binary-i386/Packages --exclude "*" receiving file list ... 1 file to consider -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages wrote 144 bytes read 118 bytes 524.00 bytes/sec total size is 127844 speedup is 487.95 asterix:/home/paulm# 4. this fails on security however: $ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include dists/testing/updates/main/binary-i386/Packages --exclude "*" receiving file list ... 0 files to consider client: nothing to do: perhaps you need to specify some filenames or the --recursive option? Case 4 seems to apply for http.us.debian.org too. I've only seen non-us work. Cheers, Paul -- Paul Makepeace ....................................... http://paulm.com/ "If I thought it would help, then you can place a hammer in a cake." -- http://paulm.com/toys/surrealism/
Dave Dykstra
2002-Jul-22 11:11 UTC
Pipelining arbitrary file transfers + --{in,ex}clude oddness
Yes, doing explicit includes followed by exclude '*' is what people do. Because you exclude '*' and the rsync algorithm is recursive, you need to explicitly include all parent directories of the files you want to copy as well. We have discussed making a --files-from option with which people can explicitly list only the files they want to copy, and I had promised to write it, but I haven't been able to find the time. - Dave Dykstra On Sat, Jul 13, 2002 at 02:43:12AM +0100, Paul Makepeace wrote:> I'd like to rsync a series of files at arbitrary points in a tree in one > connection and am struggling to figure out how to do this. rsync seems > AFAICS to have a fundamental src_file dst_file model, optionally with > recursion. > > The best I could come up with was recursing plus multiple --include with > a final --exclude "*". The idea is to avoid multiple calls to rsync and > thus TCP/session overhead costs. Is this reasonable? A better way? > (Sneaking feeling I'm missing something really obvious at 2am, despite > looking in the FAQ, man page, google and the wishlist...) > > Anyway, doesn't this scheme make the server work hard building a > file list at its end? > > ** > > In the course of playing with this I found something I can't explain. > On one server, this scheme seems to work fine but another says there's > nothing there. If I try to get the file directly it's reported there. In > other words, (missing final --exclude "*" for clarity) > > rsync -a rsync://server/module --include path/file # "nothing to do" > rsync -a rsync://server/module/path/file # shows file & path info > > Here's a transcript to demonstrate this: (yes, I'm playing with apt-get > update speed-ups but perhaps ignore that for now :-) > Using: rsync version 2.5.6cvs protocol version 26 > > 1. non-us seems to work fine: > > $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages > receiving file list ... > 6 files to consider > drwxrwsr-x 4096 2002/07/12 01:26:18 dists > lrwxrwxrwx 3 2001/02/07 13:59:15 dists/testing -> woody > drwxrwsr-x 4096 2002/07/12 19:53:44 dists/testing/non-US > drwxrwsr-x 4096 2001/01/17 23:52:49 dists/testing/non-US/main > drwxrwsr-x 4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386 > -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages > wrote 133 bytes read 200 bytes 666.00 bytes/sec > total size is 127847 speedup is 383.92 > > 2. security ditto: > > $ rsync -Rzav --progress rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages > receiving file list ... > 6 files to consider > drwxrwsr-x 4096 2002/05/01 13:14:24 dists > lrwxrwxrwx 5 2002/05/10 02:50:17 dists/testing -> woody > drwxrwsr-x 4096 2002/07/12 21:40:06 dists/testing/updates > drwxrwsr-x 4096 2002/05/01 13:14:13 dists/testing/updates/main > drwxrwsr-x 4096 2002/07/12 21:40:00 dists/testing/updates/main/binary-i386 > -rw-rw-r-- 88472 2002/07/12 21:22:39 dists/testing/updates/main/binary-i386/Packages > wrote 137 bytes read 199 bytes 224.00 bytes/sec > total size is 88477 speedup is 263.32 > > 3. back to non-us using the --include path/file --exclude "*" hack. > All's well: > > $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include dists/testing/non-US/main/binary-i386/Packages --exclude "*" > receiving file list ... > 1 file to consider > -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages > wrote 144 bytes read 118 bytes 524.00 bytes/sec > total size is 127844 speedup is 487.95 > asterix:/home/paulm# > > 4. this fails on security however: > > $ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include dists/testing/updates/main/binary-i386/Packages --exclude "*" > receiving file list ... > 0 files to consider > client: nothing to do: perhaps you need to specify some filenames or the --recursive option? > > Case 4 seems to apply for http.us.debian.org too. I've only seen non-us > work. > > Cheers, > Paul > > -- > Paul Makepeace ....................................... http://paulm.com/ > > "If I thought it would help, then you can place a hammer in a cake." > -- http://paulm.com/toys/surrealism/ > > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Reasonably Related Threads
- Broken rsync mirroring for PuppetLabs APT repo
- Samba Permissions Problem
- point n print driver deployment for canon ip7250
- Samba4 not able to write to group writeable folder???
- User executional bit set when creating/modifying file on linux server from linux client