rsync - Oct 2012 - Maximum number of entries in a filter file

We are using rsync to maintain a backup of some relatively large file
systems, around 3 million files, but in an effort to save space have decided
that some files should not be synchronised.  In order to do this we have
created a filter file containing the list of files to exclude.  

 

This filter file is created on what is ultimately the destination end of the
synchronisation so we first copy this file to the source end.

 

rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable
/mnt/backup/audio/mp3_64/.rsync-exclude
10.0.60.64:/mnt/encoded/audio/mp3_64/

 

Then we perform the main synchronisation from source to destination

 

rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable
--filter="merge,e /mnt/backup/audio/mp3_64/.rsync-exclude"
10.0.60.64:/mnt/encoded/audio/mp3_64/ /mnt/backup/audio/mp3_64/

 

I have noticed that the .rsync-exclude file does not seem to be processed.
I see no errors related to this in the output but believe it is because it
contains over 120,000 entries and is maybe exceeding some internal 16bit
limit in rsync of the number of entries that can be in a filter file.  We
have other file systems where the filter file has around 30,000 entries and
this seems to be working fine.  

 

I was wondering:

 

Does anyone knows if this is a hard internal limit that we have hit?

 

If so, Is it possible to get around this limit by specifying more than one
file, not per directory files, just multiple files?

 

If not, I guess that moving to per directory filter files would be the
answer, we have around 5,000 directories on the file system.  The reason we
chose a single file initially was that it is very quick to synchronise this
one file.  If I do use per directory files, is there a way to synchronise
those files from destination to source quickly without checking all the
other files on the destination end?

 

For instance running this takes an awfully long time as it searches through
every dri & file for the .rsync-exclude files

rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable
--dry-run --progress --filter="include, .rsync-exclude"
--filter="include,
*/" --filter="exclude, *" /mnt/backup/audio/mp3_64-CLIP/
10.0.60.64:/mnt/encoded/audio/mp3_clip/

 

This is the verison of rsync being used:

 

rsync  version 3.0.6  protocol version 30

Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.

Web site: http://rsync.samba.org/

Capabilities:

    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,

    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

    append, ACLs, xattrs, iconv, no symtimes

 

Thanks in advance for any help.

 

Regards

Ben

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20121012/5745c5e5/attachment.html>

In <180601cda863$495c0b10$dc142130$@juno.co.uk>, on 10/12/12
   at 11:21 AM, "Ben Yarwood" <ben.yarwood at juno.co.uk> said:

Hi,
>I have noticed that the .rsync-exclude file does not seem to be
>processed. I see no errors related to this in the output but believe it
>is because it contains over 120,000 entries and is maybe exceeding some
>internal 16bit limit in rsync of the number of entries that can be in a
>filter file.
It's possible, but unlikely.  On a 32-bit platform, integers are going to
be 32-bit numbers and rsync uses these for the vast majority of the code.

I would suspect the format of the exclude patterns.  Perhaps they are not
relative to the source path.

If there is a limit other than available memory, it is unintentional.
>rsync  version 3.0.6  protocol version 30
>Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
This build is a bit old, but I don't recall any issues with merge
patterns.

Regards,

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------