rsync - Sep 2011 - protect rule prevents file sending when in merge filter file?

Hi,

We upgraded from rsync 2.6.8 to 3.0.6, and ran into a problem with P (protect)
rules coming in from filter files. The rule works if given on command line, but
if given via . (merge), it no longer works. The effect is that files are not
sent to the destination any more.

I verified the problem still exists in 3.0.9pre2. I don't think this is bug
8309 (https://bugzilla.samba.org/show_bug.cgi?id=8309) as all this happens on
a single machine.

Here's an example how to set up to replicate this problem. We want to copy
from
'aaa' to 'bbb' any files which exist in 'aaa', but leave
untouched any other
files in 'bbb'.

 $ cd /tmp
 $ mkdir aaa bbb aaa/x mkdir bbb/x
 $ touch aaa/x/file aaa/file bbb/x/y
 $ echo 'P /x/***' > my.rules

Expected behaviour:

 $ rsync -nzavmH --delete-excluded -f '. my.rules' $PWD/aaa/ $PWD/bbb/
 building file list ... done
 ./
 file
 x/file

 sent 126 bytes  received 38 bytes  328.00 bytes/sec
 total size is 0  speedup is 0.00

Behaviour in rsync 3.0.6 / 3.0.9pre2:

 $ rsync -nzavmH --delete-excluded -f '. my.rules' $PWD/aaa/ $PWD/bbb/
 building file list ... done
 ./
 file

 sent 60 bytes  received 18 bytes  156.00 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)

Putting the 'P' filter directly on command line works:

 $ rsync -nzavmH --delete-excluded -f 'P /x/***' $PWD/aaa/ $PWD/bbb/ 
 building file list ... done
 ./
 file
 x/file

 sent 88 bytes  received 21 bytes  218.00 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)

Apparently in this trivial example this can be worked around by using
".,r",
which I find a bit confusing since "P" is a receiver side option. At
any rate
I am not sure it's a general fix for our complete rule sets.

 $ rsync -nzavmH --delete-excluded -f '.,r my.rules' $PWD/aaa/ $PWD/bbb/
 building file list ... done
 ./
 file
 x/file

 sent 88 bytes  received 21 bytes  218.00 bytes/sec
 total size is 0  speedup is 0.00 (DRY RUN)

Is this a bug? Looking at debug output from running with -vvvv, the main
difference between -f 'P ...' and -f '. ...' is that former
generates (a)
and latter generates (b). Note the rule changes from '-r' to
'-sr', and
the sending side files are removed, which explains the behaviour seen.

Whether it's a bug, could anyone suggest an alternative rule set which
would allow us to achieve our goal here so we can work with 3.0.x rsync?

Regards,
Lassi

(a)

[client] add_rule(-r /x/***)
FILE_STRUCT_LEN=24, EXTRA_LEN=4
cmd=<NULL> machine=<NULL> user=<NULL> path=/tmp/bbb/
cmd[0]=. cmd[1]=/tmp/bbb/ 
note: iconv_open("UTF-8", "UTF-8") succeeded.
(Client) Protocol versions: remote=30, negotiated=30
(Server) Protocol versions: remote=30, negotiated=30
building file list ... 
[sender] change_dir(/tmp/aaa)
[sender] make_file(.,*,0)
[sender] make_file(file,*,2)
[sender] make_file(x,*,2)
[sender] make_file(x/file,*,2)
done

(b)

[client] parse_filter_file(my.rules,10400,1)
[client] add_rule(-sr /x/***)
FILE_STRUCT_LEN=24, EXTRA_LEN=4
cmd=<NULL> machine=<NULL> user=<NULL> path=/tmp/bbb/
cmd[0]=. cmd[1]=/tmp/bbb/ 
note: iconv_open("UTF-8", "UTF-8") succeeded.
(Client) Protocol versions: remote=30, negotiated=30
(Server) Protocol versions: remote=30, negotiated=30
building file list ... 
[sender] change_dir(/tmp/aaa)
[sender] make_file(.,*,0)
[sender] make_file(file,*,2)
[sender] hiding directory x because of pattern /x/***
done

On Tue, Sep 13, 2011 at 12:09 PM, Lassi Tuura <lat at cern.ch> wrote:
> Whether it's a bug, could anyone suggest an alternative rule set
> which would allow us to achieve our goal here so we can work with 3.0.x
> rsync?
>
The problem is your use of --delete-excluded, which is an old option from
the days prior to filter files.  The way that --delete-excluded is
implemented, it adds an 's' (sender-side) modifier to all the
include/exclude/filter options that you specify on the command-line (unless
the command-line filter option has an explicit side specified).  Thus
--exclude=foo turns into --filter='-s foo' and --exclude-from=FILE turns
into a (relaxed-prefix) --filter="merge,s FILE" rule.  In 3.0.x, the
filter
code treats the specifying of a protect rule inside a server-side rule file
as though the receiver-side option gets added to the server-side option,
turning it into a normal exclude (i.e. "P foo", aka "-r foo"
turns into "-sr
foo").  In the 3.1.0dev code, your test case errors out with a rule
conflict
(the specifying of a receiver-side rule inside a sender-side rule file).
 Specifying the "-r /x/***" rule on the command-line isn't
considered a
conflict, since it never gets the implied 's' modifier added to it.

For the future, I'm thinking that it would probably be better to just allow
the specified-side rule to override an specified-side merge rule, which
would make the merge,s setting just a default to create sender-side rules
rather than it requiring all the rules therein to not specify a side.
 However, care needs to be taken to ensure that the side optimizations (and
per-dir merge files) are all handled correctly.

Your best bet for the meantime is to not mix --delete-excluded with
side-specifying filter rules, and just specify your hide rules directly --
i.e. replace --exclude-from=FILE with --filter=".s FILE" (though every
exclude line would need to have a "- " prefix) or --filter=".-s
FILE"
(though every line would need to NOT have a +/- prefix), etc.

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20110913/f716b486/attachment.html>