I've worked up a new release of my --files-from patch. It has the
following changes from the last release:
New Changes:
===========
- The -R option is now on by default when --files-from has been
specified. The user can specify --no-relative to turn this off.
- The -a (--archive) option does not imply -r when --files-from was
specified. The user can still specify it manually, though.
- Implied directories are sent unless the --no-implied-dirs option is
specified. Bug fixes in the implied-dir code have also been applied
to make it more efficient and to properly remove duplicates.
- You can now specify a remote file-list to read by using a "host:"
(or
just ':') prefix. The host MUST match the host of one end of the
transfer (or be empty). E.g.:
rsync -av --files-from=remote:/path/file-list remote:/from /to
rsync -av --files-from=:/path/file-list /from remote:/to
rsync -av --files-from=:file-list remote::module/from /to
The daemon version sanitizes the files-from value to ensure that it
exists in the referenced module, but the remote-shell version can
refer to the file anywhere in the file system.
- I also included some gratuitous option-code cleanup/simplification.
Previously Released Changes:
===========================
- All paths read by --files-from are sanitized so that they aren't
absolute and can't specify enough "../" to break out of the
starting
dir (that was specified on the command line).
- Various bug fixes.
What Is the --files-from Option?
===============================
This option allows the user to specify an explicit list of files to
transfer, much like could be done on the command-line, but without an
overly restrictive limit on how many files can be specified. This can
be much more efficient than doing a recursive directory scan using
include & exclude rules, and enables more complex selection criteria
than can be done by name-matching alone (since the file list is created
by whatever means you desire).
It might help to visualize this new command:
rsync -a --files-from=/some/list /source /dest
as being roughly equivalent to this old command (note the lack of -r and
the presence of -R -- it's not a mistake):
cd /source
rsync -lptgoD -R `cat /some/list` /dest
With these exceptions:
- Directories mentioned in the "list" file are created on the
destination (without transferring their contents unless -r is
specified).
- Paths in the "list" are sanitized to ensure that they don't
refer
anywhere outside of the default ("/source") dir.
- The number of files is not limited by the command-line character
limit.
- You can specify a list of files separated by nulls (instead of
newlines) if you add the --null option.
Where To Get It:
===============
http://www.clari.net/~wayne/rsync-files-from.patch
** This code is still early in its life, and should be used with caution. **
Please help me test it and feel free to let me know what you think.
Enjoy!
..wayne..
On Fri, Jan 17, 2003 at 07:22:00PM -0800, Wayne Davison wrote:> I've worked up a new release of my --files-from patch. It has the > following changes from the last release: > > New Changes: > ===========> > - The -R option is now on by default when --files-from has been > specified. The user can specify --no-relative to turn this off. > > - The -a (--archive) option does not imply -r when --files-from was > specified. The user can still specify it manually, though. > > - Implied directories are sent unless the --no-implied-dirs option is > specified. Bug fixes in the implied-dir code have also been applied > to make it more efficient and to properly remove duplicates. > > - You can now specify a remote file-list to read by using a "host:" (or > just ':') prefix. The host MUST match the host of one end of the > transfer (or be empty). E.g.: > > rsync -av --files-from=remote:/path/file-list remote:/from /to > rsync -av --files-from=:/path/file-list /from remote:/to > rsync -av --files-from=:file-list remote::module/from /to > > The daemon version sanitizes the files-from value to ensure that it > exists in the referenced module, but the remote-shell version can > refer to the file anywhere in the file system.This is a very nice touch. I would have required the file list to be local to the client but you've taken a spendid aproach.> - I also included some gratuitous option-code cleanup/simplification. > > Previously Released Changes: > ===========================> > - All paths read by --files-from are sanitized so that they aren't > absolute and can't specify enough "../" to break out of the starting > dir (that was specified on the command line). > > - Various bug fixes. > > What Is the --files-from Option? > ===============================> > This option allows the user to specify an explicit list of files to > transfer, much like could be done on the command-line, but without an > overly restrictive limit on how many files can be specified. This can > be much more efficient than doing a recursive directory scan using > include & exclude rules, and enables more complex selection criteria > than can be done by name-matching alone (since the file list is created > by whatever means you desire). > > It might help to visualize this new command: > > rsync -a --files-from=/some/list /source /dest > > as being roughly equivalent to this old command (note the lack of -r and > the presence of -R -- it's not a mistake): > > cd /source > rsync -lptgoD -R `cat /some/list` /dest > > With these exceptions: > > - Directories mentioned in the "list" file are created on the > destination (without transferring their contents unless -r is > specified). > > - Paths in the "list" are sanitized to ensure that they don't refer > anywhere outside of the default ("/source") dir. > > - The number of files is not limited by the command-line character > limit. > > - You can specify a list of files separated by nulls (instead of > newlines) if you add the --null option. > > Where To Get It: > ===============> > http://www.clari.net/~wayne/rsync-files-from.patch > > ** This code is still early in its life, and should be used with caution. ** > Please help me test it and feel free to let me know what you think. > > Enjoy!I've pulled the patch down and will give it a whirl. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
On Fri, Jan 17, 2003 at 07:22:00PM -0800, Wayne Davison wrote:> I've worked up a new release of my --files-from patch. It has the > following changes from the last release: > > New Changes: > ===========> > - You can specify a list of files separated by nulls (instead of > newlines) if you add the --null option.Just a thought here... I've been a bit bothered by the --null option, firstly in terms of option proliferation which is bad enough with this patch. Mainly because we have several options that accept multiple values from a file and we aren't switching to null termination for them but --null is rather unspecific. Unfortunately, i think the functionality is needed but can't think of a way of reliable auto-detection. Try this on for size. Instead of being orthogonal with tar, cpio and xargs, be orthogonal with find. Like -print0 use --files-from0 to specify that that the file list is null terminated. In that way --files-from and --files-from0 will be a single manpage entry and it will be clear that the null termination only applies to the file-list even without cross-referencing. It also means the user only needs one command-line option for the file list regardless of entry termination. It also occurs to me that it might make sense to always transfer the file-list over the network null terminated. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt