rsync - Mar 2008 - DO NOT REPLY [Bug 5324] New: with option --xattrs the process rsync is more long time

https://bugzilla.samba.org/show_bug.cgi?id=5324

           Summary: with option --xattrs the process rsync is more long time
           Product: rsync
           Version: 3.0.0
          Platform: x86
        OS/Version: Other
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: fauthier@free.fr
         QAContact: rsync-qa@samba.org


system on both server :
os x server 10.4.11

when i use the option -X (xattr) to synchronize about 400000 files between two
server, the time is four time more long that i don't use this (2h us 1/2h).

syntaxe : 
rsync -aAX --del --force /source/ server2:/dest/ (400000 files -> 2hours)
rsync -aA --del --force /source/ server2:/dest/ (400000 files -> half hours)

thank you
F.A.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5324


matt@mattmccutchen.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement
            Summary|with option --xattrs the    |Reduce the performance
                   |process rsync is more long  |penalty of --xattrs on Mac
                   |time                        |OS X




------- Comment #1 from matt@mattmccutchen.net  2008-03-14 12:41 CST -------
Asking rsync to do more (preserve the xattrs) will inevitably make it take
longer, but a 4x slowdown does seem excessive.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5324


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED




------- Comment #2 from wayned@samba.org  2008-03-15 03:04 CST -------
Do your files have a lot of differing xattrs?  One thing I never much liked
about the xattrs patch (from the very beginning) is that the code attempts to
do a very simplistic linear search through all the prior xattrs looking for a
matching set of attributes (to share matching attributes between files).  If
your files have a lot of xattr entries, that search will eat up more and more
time as the list of unique attributes grows.

One solution to this might be to create a hash of all the names and xattr data,
and then store the xattrs in a hash lookup.  That should speed things up quite
a bit when the list of unique xattr values grows large.

Note:  if you are using the osx-create-time.diff, please switch to the
crtime.diff instead -- the oxs-create-time.diff patch is known to be slow,
quitely possibly due to the bloating of unique xattr values in the list.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5324





------- Comment #3 from fauthier@free.fr  2008-04-10 11:35 CST
-------
> One solution to this might be to create a hash of all the names and xattr
data,
and then store the xattrs in a hash lookup.  That should speed things up quite
a bit when the list of unique xattr values grows large.

Thank you,
but how do it this hash ? 
thank's a lot

(I use now the rsync 3.0.2)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5324


mike@bombich.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mike@bombich.com




------- Comment #4 from mike@bombich.com  2009-06-20 21:53 CST -------
Mac OS X makes extensive use of xattrs, and I've seen hundreds of thousands
of
unique xattrs on several end-user systems.  In those cases rsync eventually
runs out of memory and bails.

rsync should probably stop calling find_matching_xattr when the list reaches a
specified size.  And at least for Mac OS X, xattrs shouldn't be cached in
rsync_xal_l, they should probably be compared on the fly by the generator
somewhere near generate_and_send_sums.

Also, to address Fabrice's concern, performance will ultimately be directly
linked to the lack of a quick heuristic for determining whether xattrs have
been modified.  xattrs don't have modifications dates, so you can use either
the size or a checksum to determine if they've changed.  In my analysis,
rsync
spent 20% of its time in md5_process on a task involving many xattrs with lots
of data.  Unless we come up with something faster than md5 (potentially at the
cost of reliability), this is just a performance hit we'll have to live
with.

I'm going to take a stab at this, but I'm curious whether you've put
any more
thought into how xattr support is implemented since your last comment here.

Thanks!
Mike Bombich


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5324

--- Comment #5 from Björn Jacke <bjacke at samba.org> ---
how about adding an option like "--use-ctime-before-xattr-compares",
which only
reads and compares EAs for files where the ctime on the source side is newer
than the ctime on the target side. EA modifications update the ctime ususally.
This would be a way to speed up syncing with EAs quite a lot I think.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

--- Comment #6 from Stefan Metzmacher <metze at samba.org> ---
Created attachment 12285
  --> https://bugzilla.samba.org/attachment.cgi?id=12285&action=edit
A first work in progress patch to add a hashtable

I need to clean this up and do more tests.

But with 1000 unique xattrs I got 50% less cpu instruction in callgrind.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

Stefan Metzmacher <metze at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #12285|0                           |1
        is obsolete|                            |

--- Comment #7 from Stefan Metzmacher <metze at samba.org> ---
Created attachment 12286
  --> https://bugzilla.samba.org/attachment.cgi?id=12286&action=edit
valgrind clean work in progress patch

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

Stefan Metzmacher <metze at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
  Attachment #12286|0                           |1
        is obsolete|                            |
                 CC|                            |wayned at samba.org
  Attachment #12287|                            |review?(wayned at samba.org)
              Flags|                            |

--- Comment #8 from Stefan Metzmacher <metze at samba.org> ---
Created attachment 12287
  --> https://bugzilla.samba.org/attachment.cgi?id=12287&action=edit
Possible patch for master

I hope this is an acceptable patchset to fix the problem for rsync master.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|3.0.0                       |3.1.3
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Wayne Davison <wayned at samba.org> ---
The patchset looks very nice. Thanks!

I've made some very minor tweaks and committed it.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

--- Comment #10 from Oren Kishon <orenk at ctera.com>
---
> how about adding an option like
"--use-ctime-before-xattr-compares", which only > reads and
compares EAs for files where the ctime on the source side is newer
> than the ctime on the target side. EA modifications update the ctime
ususally. > This would be a way to speed up syncing with EAs quite a lot I
think.
Has anyone started developing this?

Thanks you
Oren Kishon, Ctera

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=5324

--- Comment #11 from Björn Jacke <bjacke at samba.org> ---
no, because the problem was fixed with metzes patch. if you see a need for the
ctime compare, then please open a new bug report for it. this bug here is
closed and fixed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.