rsync - Jul 2015 - cut-off time for rsync ?

You could use find to build a filter to use with rsync, then update the 
filter every few days if it takes too long to create.

I have used a script to build a filter on the source server to exclude 
anything over 5 days old, invoked when the sync starts, but it only 
parses around 2000 files per run.

Mark.


On 2/07/2015 2:34 a.m., Ken Chase wrote:
> What is taking time, scanning inodes on the destination, or recopying the
entire
> backup because of either source read speed, target write speed or a slow
interconnect
> between them?
>
> Do you keep a full new backup every day, or are you just overwriting the
target
> directory?
>
> /kc
>
>
> On Wed, Jul 01, 2015 at 10:06:57AM +0200, Dirk van Deun said:
>    >> If your goal is to reduce storage, and scanning inodes doesnt
matter,
>    >> use --link-dest for targets. However, that'll keep a backup
for every
>    >> time that you run it, by link-desting yesterday's copy.
>    >
>    >The goal was not to reduce storage, it was to reduce work.  A full
>    >rsync takes more than the whole night, and the destination server is
>    >almost unusable for anything else when it is doing its rsyncs.  I
>    >am sorry if this was unclear.  I just want to give rsync a hint that
>    >comparing files and directories that are older than one week on
>    >the source side is a waste of time and effort, as the rsync is done
>    >every day, so they can safely be assumed to be in sync already.
>    >
>    >Dirk van Deun
>    >--
>    >Ceterum censeo Redmond delendum
>

On Thu, 02 Jul 2015 20:57:06 +1200, Mark wrote:
> You could use find to build a filter to use with rsync, then update the
> filter every few days if it takes too long to create.
If you're going to do something of that sort, you might want instead to 
consider truly tracking changes.  This catches operations that find will 
miss, such as deletes, renames, copies preserving timestamp ("cp -
p ..."), and probably other operations not coming to mind at the moment.

Look at tools like inotifywait, auditd, or kfsmd to see what's easily 
available to you and what best fits your needs.

[Though I'd also be surprised if nobody has fed audit information into 
rsync before; your need doesn't seem all that unusual given ever-growing 
disk storage.]

In addition to catching operations that a find would miss, this also 
avoids the cost of scanning file systems which is the immediate need 
being discussed.  On the other hand, this isn't free either.  I imagine 
that there's some crossover point on one side of which scanning is better 
and on the other auditing is better.

	- Andrew

On Mon, 13 Jul 2015 02:19:23 +0000, Andrew Gideon wrote:
> Look at tools like inotifywait, auditd, or kfsmd to see what's easily
> available to you and what best fits your needs.
> 
> [Though I'd also be surprised if nobody has fed audit information into
> rsync before; your need doesn't seem all that unusual given
ever-growing
> disk storage.]
I wanted to take this a bit further.  I've thought, on and off, about 
this for a while and I always get stuck.

I use rsync with --link-desk as a backup tool.  For various reasons, this 
is not something I want to give up.  But, esp. for some very large file 
systems, doing something that avoids the scan would be desirable.

I should also add that I mistrust time-stamp, and even time-stamp+file-
size, mechanism for detecting changes.  Checksums, on the other hand, are 
prohibitively expensive for backup of large file systems.

These both bring me to the idea of using some file system auditing 
mechanism to drive - perhaps with an --include-from or --files-from - 
what rsync moves.

Where I get stuck is that I cannot envision how I can provide rsync with 
a limited list of files to move that doesn't deny the benefit of --link-
dest: a complete snapshot of the old file system via [hard] links into a 
prior snapshot for those files that are unchanged.

Has anyone done something of this sort?  I'd thought of preceding the 
rsync with a "cp -Rl" on the destination from the old snapshot to the
new
snapshot, but I still think that this will break in the face of hard 
links (to a file not in the --files-from list) or a change to file 
attributes (ie. a chmod would effect the copy of a file in the old 
snapshot).

Thanks...

	Andrew