rsync - Aug 2008 - DO NOT REPLY [Bug 5728] New: Rsync --min-size and --max-size filter too late in the process.

https://bugzilla.samba.org/show_bug.cgi?id=5728

           Summary: Rsync --min-size and --max-size filter too late in the
                    process.
           Product: rsync
           Version: 2.6.9
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: minor
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: r.e.wolff@bitwizard.nl
         QAContact: rsync-qa@samba.org


To allow me to copy a very large directory that cannot be copied in one go (see
bug 5727) I restricted the sizes of the files to part of the range of
filesizes.

However, it turns out that rsync still loads all files into the in-memory list
of files to copy. This means that rsync still crashes even when I restrict the
number of files-to-copy by 95%. 

(Copying by subdirectory doesn't work for me: there are loads of hardlinks,
and
because I know that files of size XX won't be hardlinked to files of size
YY, I
can copy in stages by limiting the filesizes).


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5728


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX
            Summary|Rsync --min-size and --max- |Rsync --min-size and --max-
                   |size filter too late in the |size filter too late in the
                   |process.                    |process.




------- Comment #1 from wayned@samba.org  2008-08-31 12:29 CST -------
Those options just affect what files in the file-list may be transferred.  If
they actually excluded files from the file-list, the receiving side would not
be able to correctly handle a --delete option.

You should instead sub-set the directory using an exclude, such as
--exclude='dir/[a-m]*' for one run, and --exclude='dir/[n-z]*'
for the other
(or something similar).


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5728


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|minor                       |enhancement
             Status|RESOLVED                    |REOPENED
         Resolution|WONTFIX                     |
            Summary|Rsync --min-size and --max- |Change --min-size and --max-
                   |size filter too late in the |size to filter the file
                   |process.                    |lists, like excludes
            Version|2.6.9                       |3.1.0




------- Comment #2 from wayned@samba.org  2008-08-31 13:31 CST -------
Actually, I was too hasty in my closure.  Filtering the file lists would work
OK as long as the receiver was doing a similar filter.  Delete would behave
differently than with the current options, but not-deleting certain
out-of-range files would be comparable to the exclusions that happen for the
current filter rules.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5728





------- Comment #3 from r.e.wolff@bitwizard.nl  2008-08-31 16:18 CST -------
I do not want to delete files on the recieving side. So there is no need to
"prepare" for that possibility. 

And as I said, there are enormous amounts of hardlinks between the directories,
that would be broken if I copied per-directory. 

There could be something like 
   if (otherside_needs_full_list || filter (curpath)) 
       send_this_file (curpath); 
in the code that sends the file list?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.