rsync - Feb 2017 - [Bug 12569] New: Missing directory errors not ignored

https://bugzilla.samba.org/show_bug.cgi?id=12569

            Bug ID: 12569
           Summary: Missing directory errors not ignored
           Product: rsync
           Version: 3.1.2
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: axkibe at gmail.com
        QA Contact: rsync-qa at samba.org

When using --delete-missing-args and --ignore-errors with a a file list it
terminates anyway if one directory, that is not a sub of root, is not existing.

See:

~$ mkdir -p src/a trg/a
~$ echo "/a/b/c" > list
~$ /usr/local/bin/rsync -slt --ignore-errors --force --ignore-missing-args
--delete-missing-args --files-from=list --rsync-path=/usr/local/bin/rsync src
localhost:`pwd`/trg
file has vanished: "/home/axel/src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270)
[generator=3.1.2]
~$ /usr/local/bin/rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.


Following patch fixes it for me, albeit it ignores this kind of error now
always:
--- rsync-3.1.2-org/generator.c    2015-12-05 20:10:24.000000000 +0100
+++ rsync-3.1.2/generator.c    2017-02-07 10:58:33.768453242 +0100
@@ -1264,10 +1264,11 @@
              && (*dn != '.' || dn[1]) /* Avoid an issue with
--relative and
the "." dir. */
              && (prior_dir_file && strcmp(dn,
f_name(prior_dir_file, NULL)) !0)
              && flist_find_name(cur_flist, dn, 1) < 0) {
+                io_error |= IOERR_VANISHED;
                 rprintf(FERROR,
-                    "ABORTING due to invalid path from sender:
%s/%s\n",
+                    "WARNING: invalid path from sender: %s/%s\n",
                     dn, file->basename);
-                exit_cleanup(RERR_PROTOCOL);
+                return;
             }
             if (relative_paths && !implied_dirs
              && do_stat(dn, &sx.st) < 0) {

Now the exit code is 24, partial transfer, as would be expected.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #1 from Marc Krämer <mageia at mokraemer.de> ---
I'd like to point out that this change is a changed behavior that breaks
some
scripts depending on this behavior.
Can you consider to change it to the original behavior, or add a new parameter,
that causes missing directories to be ignored.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #2 from Axel Kittenberger <axkibe at gmail.com> ---
shouldn't "--ignore-errors" already be that parameter?
It uses --force also.

--ignore-errors --force --i-really-mean-it? :)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #3 from Marc Krämer <mageia at mokraemer.de> ---
that is my understanding too! And this was true before the last release.

Basic tools like rsync should not break their behaviour!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #4 from Marc Krämer <mageia at mokraemer.de> ---
My bugreport at mageia: https://bugs.mageia.org/show_bug.cgi?id=21395

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #5 from Dave Gordon <dg32768 at zoho.eu> ---
I think you need to add "--no-implied-dirs" to get the behaviour you
want.

The issue is that the contents list contains /a/b/c, so problems with that
specific file are suppressed by "--ignore-missing-args", but /a/b is
not
a missing argument, it's a missing implied directory. Adding the extra flag
means that it won't be (implicitly) transferred, but would instead be
created
on the destination if necessary. Then you just get a warning about the file
that's vanished:

$ rsync -slt --delete-missing-args --files-from=list src trg
file has vanished: ".../src/a/b"
ABORTING due to invalid path from sender: a/b/c
rsync error: protocol incompatibility (code 2) at generator.c(1270)
[generator=3.1.1]

$ rsync -slt --delete-missing-args --no-implied-dirs --files-from=list src trg
file has vanished: ".../src/a/b"
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1183) [sender=3.1.1]

HTH,
.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #6 from Marc Krämer <mageia at mokraemer.de> ---
ups, didn't get a notice from your reply.

Thanks for your explanation. This was not obvious to me. It should be
documented, the behaviour has changed.

You can close this one, thanks.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #7 from Axel Kittenberger <axkibe at gmail.com> ---
No please don't close. Still not the behavior I'd expect:

"""
~$ mkdir test
~$ cd test
test$ mkdir -p src/a trg/a
test$ echo "/a/b/c" > list
test$ /usr/bin/rsync -slt --ignore-errors --force --ignore-missing-args
--delete-missing-args --files-from=list --no-implied-dirs src
localhost:`pwd`/trg
file has vanished: "/home/axel/test/src/a/b"
rsync warning: some files vanished before they could be transferred (code 24)
at main.c(1196) [sender=3.1.2]
test$ diff -r src/ trg/
Only in trg/a: b
"""

Unless I misunderstand something. It still should not create b at the target,
if it isn't in the source.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #8 from Marc Krämer <mageia at mokraemer.de> ---
@Axel: you're right. This is not what we want. Even the output 
sync warning: some files vanished before they could be transferred
is not desireable if the parameter is called "ignore" there should not
be any
output.
And the return code of this transfer should not indicate any errors.

I assume, you use this for automatic synchronisation between servers as well? I
don't understand why they changed the behaviour, before this change, it
worked
as desired.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #9 from Axel Kittenberger <axkibe at gmail.com> ---
@Marc, indeed. I'm the author of Lsyncd.

https://github.com/axkibe/lsyncd

If this could work properly, it would simplify things a lot, also improve
perfomance a good deal. Due to this bug I had to drop the use of --files-from
feature and keep creating include/exclude filters on the fly, which also means
rsync has to inspect way more files, which is a quite a penalty with
directories and a lot files.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #10 from Marc Krämer <mageia at mokraemer.de> ---
@Axel: cool, I've played a bit with your tool, but for my needs with many
directories inotify was the pitfall.

I'm coauthor on sfs (https://github.com/mokraemer/sfs) which uses fuse for
signaling. And then, as you do, rsync for synchronization. So we have both the
same problem here :(

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

--- Comment #11 from Paul Slootman <paul at debian.org> ---
Created attachment 14775
  --> https://bugzilla.samba.org/attachment.cgi?id=14775&action=edit
proposed patch

This patch is based on the patch posted earlier, but checks the options
--delete-missing-args and --ignore-errors.

I see no reason why this shouldn't be patched into the main rsync sources.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12569

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #12 from Wayne Davison <wayne at opencoder.net> ---
This check was added to try to prevent an untrusted sender from sneaking
through some bad file-list data.  However, the --delete-missing-args option
does seem to have the ability to send these incongruent lists.  I'm thinking
that the best fix is probably to have the sender fudge up the missing parent
dirs if they can be marked as missing like the missing args.  In the meantime,
I'm checking in a slightly different version of Paul's patch that makes
the
receiver just complain about the missing parent dir info and continue on (it
does not return or a valid missing arg won't be deleted).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.