rsync - Oct 2013 - [Bug 10233] New: rsync is spending a lot of time lstat64()'ing --exclude'd files

https://bugzilla.samba.org/show_bug.cgi?id=10233

           Summary: rsync is spending a lot of time lstat64()'ing
                    --exclude'd files
           Product: rsync
           Version: 3.1.0
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: darxus at chaosreigns.com
         QAContact: rsync-qa at samba.org


I ran:  rsync -Pva --exclude '*.gz' / <destination>

For hours, strace has been scrolling:

lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=74, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=419, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=449, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=408, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=75, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=579, ...}) =
0
lstat64("<file>.gz", {st_mode=S_IFREG|0644, st_size=339, ...}) =
0

Seems like this statting of files which match the --exclude pattern could be
skipped, saving a lot of time in some cases?

I'm using rsync version 3.0.7.  BasketCase in IRC reproduced this with
v3.1.0.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10233

--- Comment #1 from Darxus <darxus at chaosreigns.com> 2013-10-29 15:57:28
UTC ---
A useful workaround would be something like:

find / | grep -v 'gz$' > filelist.txt
rsync -Pva --files-from=filelist.txt / <destination>

Also, don't rsync /proc/kcore :/

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10233

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Severity|normal                      |enhancement

--- Comment #2 from Wayne Davison <wayned at samba.org> 2013-11-28
17:29:51 UTC ---
Rsync wants to know if it is a file or a directory before applying the exclude
rules, or it wouldn't have enough information for dir-only exclude rules. 
It
might be possible to make the stat call lazy, where the first dir-only rule to
match the name checks if the file-type is known, and does a stat if it is not. 
However, that might over complicate things.  I'm marking this as an
enhancement
request.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.