rsync - Nov 2010 - How to sync an exact list of files, Including deletes!?

Hello,

I'm author of Lsyncd - the live syncing deamon -
http://code.google.com/p/lsyncd/ - a daemon that uses Linux` inotify
to watch for filesystem changes - aggregates them for a few seconds
and then periodically calls rsync to transfer the changes to
target(s). Version 1 was simply aware of directories only, and it
called rsync once with --delete -d for every directory in which
anything has happened. I'm now coding Version 2, this goes further and
keeps track for every file. Beside doing this to optimize on moves, I
want to optimize performance also with its interface to rsync.

While running, Lsyncd knows exactly which files need transfer and
rsync's --files-from (with =- to get the filelist from stdin-pipe)
sounds like an ideal solution for this, also to reduce the number of
times rsync gets spawned. Coded that, now in beta stage there is a
nasty problem, rsync refuses to accept deletions that way!

It says e.g. "rsync: link_stat "/tmp/ltest.M2e/src/z/t/fq"
failed: No
such file or directory (2)" and refuses to delete the file on the
target.

What good solutions can you suggest?

I thought about instead of "--files-from=-" to use ?"--recursive
--include-from=- --exclude=*". Beside I don't know how this evaluates
performance wise, it does not quite produce the effects, at least not
easily without further bloating the file-list. I suppose for every
directory on the include list, it needs to add rules for all parent
directories, so they get included in recursiveness.

Lsyncd does support a mode where ssh access is as well available on
the target. That way it can move moved files on the target, instead of
retransmitting them like a normal incremental rsync does. In that
case, it can as well simply log in there and call rm for removed
files. But I do want to support a rsync-only mode, for example if the
target runs the rsyncd daemon proving modules.

Could we get future versions of rsync accept "--delete --files-from=-"
rule to delete files and (recursively?) directories on the target that
are in the file-list but locally not existent?

Kind regards,
Axel

In <AANLkTinDfc=z8O37KS9TCRLEqPEumYd4753f_3MFbS9M at mail.gmail.com>, on
11/20/10
   at 10:56 PM, Axel Kittenberger <axkibe at gmail.com> said:

Hi,
>While running, Lsyncd knows exactly which files need transfer and
rsync's
>--files-from (with =- to get the filelist from stdin-pipe) sounds like an
>ideal solution for this, also to reduce the number of times rsync gets
>spawned. Coded that, now in beta stage there is a nasty problem, rsync
>refuses to accept deletions that way!
What might work for the deletes is a dynamically built --exclude-from list
in combination with --delete excluded and possibly --prune-empty-dirs. 
I've never tried this with --files-from, but it should work as long as
non-existant source file does not cause a problem with the filter logic.

You should not need --recursive.  You just need the --exclude-from file to
contain full paths.


Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------

In <AANLkTinDfc=z8O37KS9TCRLEqPEumYd4753f_3MFbS9M at mail.gmail.com>, on
11/20/10
   at 10:56 PM, Axel Kittenberger <axkibe at gmail.com> said:

Hi,
>While running, Lsyncd knows exactly which files need transfer and
rsync's
>--files-from (with =- to get the filelist from stdin-pipe) sounds like an
>ideal solution for this, also to reduce the number of times rsync gets
>spawned. Coded that, now in beta stage there is a nasty problem, rsync
>refuses to accept deletions that way!
My first idea did not work, which I thought might happen, but I came up
with a reasonably cheap alternative using -dirs and filters.

Try this command line

  rsync -n -i --dirs --delete-excluded -f._filters from/ to

where filters contains

+ file1
+ file2
R deleted
P *

The file list contains only file1 and file2 so the receiver has less work
to do than without the filters.  It should not be too difficult to expand
this method to work for source files spread across multiple directories.

One downside of this approach compared to --dirs without filters is that
if something goes wrong at just the right time, the source and destination
can get permamently out of sync.  However, for directories with a large
number of files, --dirs with filters should perform better.  Perhaps the
first update should use plain --dirs.

Steven

-- 
----------------------------------------------------------------------
"Steven Levine" <steve53 at earthlink.net>  eCS/Warp/DIY etc.
www.scoug.com www.ecomstation.com
----------------------------------------------------------------------

On Sat, 2010-11-20 at 22:56 +0100, Axel Kittenberger
wrote:
> While running, Lsyncd knows exactly which files need transfer and
> rsync's --files-from (with =- to get the filelist from stdin-pipe)
> sounds like an ideal solution for this, also to reduce the number of
> times rsync gets spawned. Coded that, now in beta stage there is a
> nasty problem, rsync refuses to accept deletions that way!
> 
> It says e.g. "rsync: link_stat "/tmp/ltest.M2e/src/z/t/fq"
failed: No
> such file or directory (2)" and refuses to delete the file on the
> target.
> 
> What good solutions can you suggest?
You want the --delete-missing-args option of the current development
rsync.

-- 
Matt

And another case if the parent directory of the target also doesnt exist:

/usr/bin/rsync -r /tmp/ltest.Pkr/src//l/w/c/r/a/ /tmp/ltest.Pkr/trg//l/w/c/r/a/
rsync: change_dir "/tmp/ltest.Pkr/src//l/w/c/r/a" failed: No such file
or directory (2)
rsync: mkdir "/tmp/ltest.Pkr/trg//l/w/c/r/a" failed: No such file or
directory (2)
rsync error: error in file IO (code 11) at main.c(594) [receiver=3.0.3]
rsync: connection unexpectedly closed (96 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(635) [sender=3.0.3]

So far I guessed exit code 12 from the text to be a network error and
to keep trying it again  - which I suppose isn't a good idea, since
this will never go trough until /tmp/ltest.Pkr/trg//l/w/c exists.

On Tue, Nov 23, 2010 at 11:32 PM, Axel Kittenberger <axkibe at gmail.com>
wrote:
>> You want the --delete-missing-args option of the current development
>> rsync.
>
> Yes, I suppose I want that! :-) I hope this works with --force as well
> if there is a dir in --files-from? But futher supposing it takes quite
> a long time until I can suppose every user with medicore updateness
> has a rsync installed that can do this. For now I made it in ssh-less
> pure rsync-mode to call rsync once for with --files-from for all files
> that created/changed, and then for every directory in which stuff has
> been deleted/movedfrom.
>
> I got another question tough running extensive tests, the daemon
> happens to do this once in a while if creates/deletes/moves piling up
> faster than inotify tracks, it calls rsync for a source->destination
> pair, where the source dir no longer exists, normally would be no
> problem:
>
> $ /usr/bin/rsync -vP ?-d /tmp/ltest.WQS/src//n/c/d/
/tmp/ltest.WQS/trg//n/c/d/
> building file list ...
> rsync: change_dir "/tmp/ltest.WQS/src//n/c/d" failed: No such
file or
> directory (2)
> 0 files to consider
> created directory /tmp/ltest.WQS/trg//n/c/d
>
> I expect rsync to just fail in that case, but it creates the target
> directory nevertheless :-( Is there some parameter I can add to rsync
> so it wont create directories on the target, if there isnt a source
> directory for it?
>
> Kind regards, Axel
>