rsync - Oct 2016 - [Bug 12378] New: why i cannot exclude dir/files if using option "--files-from"

https://bugzilla.samba.org/show_bug.cgi?id=12378

            Bug ID: 12378
           Summary: why i cannot exclude dir/files if using option
                    "--files-from"
           Product: rsync
           Version: 3.1.2
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: isaja at sissa.it
        QA Contact: rsync-qa at samba.org

I don't understand if this is intended or not, but when I use
"--files-from"
and  "--exclude"  options together all files are synced anyhow.
eg.:
rsync -av -c --delete --relative --delete-excluded  --exclude=".cache"
--exclude="mydirToExclude" /home . 
it correctly doesn't copy the .cache folder.

but:
rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt
--exclude=".cache" --exclude="mydirToExclude" /home .
does copy everything.

instead using a pattern such as  --exclude="**myDirToExclude**"  
doesn't copy all lines with that pattern

myFileList.txt is something like:
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json
userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe
userName/.cache/winetricks/dotnet30/dotnetfx3.exe
userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe
userName/.cache/winetricks/msls31/InstMsiW.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html


Thanks,
alessio

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #1 from Kevin Korb <rsync at sanitarium.net> ---
It did not copy the directory you excluded it copied the files within that
directory that you explicitly told it to copy and created the appropriate
directories to allow that to happen.

IOW, .cache is not relative to
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #2 from Alessio <isaja at sissa.it> ---
(In reply to Kevin Korb from comment #1)

Excuse me, but i don't get it, what's the difference between excluding
(matching) the pattern from a --files-from  and a remote host?

debugging the rsync transfer from the remote host I get:
[sender] hiding directory home/userName/.cache because of pattern .cache

in the other case nothing is being hidden instead  but .cache  is inside my
path:
userName/.cache/winetricks/msls31/InstMsiW.exe

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #3 from Kevin Korb <rsync at sanitarium.net> ---
I didn't say anything specific about a remote host.

You excluded .cache then you told it to copy specific files some of which are
in .cache.  If there was a file named .cache inside of a directory that you
told it to copy then it would be excluded.  If you excluded
userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded. 
Exclude is not patern matched against parts of things you explicitly tell rsync
to copy...

kmk at dementia[1%]> cd /tmp
kmk at dementia[2%]> mkdir test
kmk at dementia[3%]> cd !$
cd test
kmk at dementia[4%]> mkdir src dst
kmk at dementia[5%]> cd src
kmk at dementia[6%]> mkdir -p a b c d e cache/c
kmk at dementia[7%]> cd ..
kmk at dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" >
list
kmk at dementia[9%]> cat list 
a
b
c
d
e
cache/c
kmk at dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ cache/
cd+++++++++ cache/c/
cd+++++++++ d/
cd+++++++++ e/

sent 157 bytes  received 37 bytes  388.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk at dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ d/
cd+++++++++ e/

sent 106 bytes  received 28 bytes  268.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk at dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/
dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ d/
cd+++++++++ e/

sent 121 bytes  received 31 bytes  304.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Cache isn't really being copied it is just being created to make a place for
cache/c to be stored when cache/c or c is not excluded.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|NEW                         |RESOLVED

--- Comment #4 from Wayne Davison <wayned at samba.org> ---
Command args (which includes names inside a files-from file) are never excluded
by an exclude directive. You told rsync to copy it, so it copies it. Excludes
only affect matching of files that rsync finds inside directories that you told
it to copy. It is expected that you already trimmed any unwanted files/dirs
from your args before you called rsync, since those names are under your
control.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #5 from Alessio <isaja at sissa.it> ---
(In reply to Wayne Davison from comment #4)

Well if it's by design I will not go further. But for me it's a missing
"feature" because one has to prepare the list beforehand while rsync
could do
the job "while it's there".  
It's like the impossibility to sync files newer than specific date, which I
really miss, but it's another story. Thanks for your help!


(In reply to Kevin Korb from comment #3)

Thanks for the detailed example and explanation it's exactly what happens to
me.
And if you remove the "--files-from" you see that "cache"
folder is really
excluded from the copy.
I was wondering why the behavior is different applying the
"--files-from" or
not and unfortunately WayneD said it's by design.
Thanks for your help

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12378

--- Comment #6 from Kevin Korb <rsync at sanitarium.net> ---
If you want to sync files newer than say 3 days ago that is what --files-from
is for...
cd /source
find . -mtime -3 -print | rsync -vai --files-from=- . /target

The primary purpose of --files-from is to give rsync the power of find.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.