rsync - Jul 2014 - [Bug 10724] New: rsync 3.1.1 incorrectly creates extra dirs inside --backup-dir on Mac OSX

https://bugzilla.samba.org/show_bug.cgi?id=10724

           Summary: rsync 3.1.1 incorrectly creates extra dirs inside
                    --backup-dir on Mac OSX
           Product: rsync
           Version: 3.1.1
          Platform: x86
        OS/Version: Mac OS X
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: chdiza at gmail.com
         QAContact: rsync-qa at samba.org


On MacOSX 10.9.4, new to rsync 3.1.1, I observe the incorrect creation of empty
dirs inside a --backup-dir.

To reproduce:

1. mkdir foo
2. mkdir bar
3. cd foo; touch fun
4. cd ..
5. rsync -trlvhp foo bar

Everything is as it should be: inside bar there is a clone of foo.  But now
continue (from the same working dir as from setp 5):

6. mkdir foo/newdir
7. cd foo; mv fun newdir; cd ..
8. rsync -trlvhpb --delete --backup-dir=mybackups --suffix=~ foo bar

Expected result: the directory bar should look like this:

bar/
bar/foo/
bar/foo/newdir/fun
bar/mybackups/
bar/mybackups/foo/
bar/mybackups/foo/fun~

Actual result: rsync 3.1.1 has created an empty dir "newdir" inside
mybackups.  That is, this is what bar looks like:

bar/
bar/foo/
bar/foo/newdir/fun
bar/mybackups/
bar/mybackups/foo/
bar/mybackups/foo/fun~
bar/mybackups/foo/newdir/

As you can see there is magically a "newdir" there.  It shouldn't
be
there, because there never was a "newdir" inside bar/foo prior to step
8.
Therefore, there was no "newdir" there to be "backed up"
into mybackups.

This behavior is new to 3.1.1 as best as I can tell.  I'm positive it never
occurred
on any prior version of rsync that I've ever used (2.6-ish?).

The above was just a toy example designed to illustrate the issue.  In
practice, rsync 3.1.1 left dozens of such "ghost" directories inside
my
--backup-dir.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #1 from chdiza at gmail.com 2014-07-18 03:28:52 UTC ---
My apologies, this bug is also present on rsync 3.1.0.

I still think it's a bug.  Also, I was almost right; it didn't work like
this
on 3.0.9 and earlier.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #2 from chdiza at gmail.com 2014-07-18 04:26:08 UTC ---
For fun I cloned the git repo and bisected.  The first commit that exhibits the
aforementioned buggy behavior is: 3696674bc62b0c1250027dbeedafdd7ebafdcf8b

which carries the description "More backup improvements".

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

I have a problem similar to this bug, not sure if it is the same bug or 
not, the consequences
in my case are killing off a daily snapshot mechanism.

As mentioned in the "Concern rsync failing to find attributes..."
thread,
I am doing an rsync from A->B while excluding files
in a --compare-dest base dir.

The problem is on a relatively small snapshot space is tighter.  It's
creating a 680MB fs, for a 550MB snapshot.  Then I copy the 550MB
snapshot to the 680MB fs and run out of space because
rsync created 310 thousand empty directories on the target.

If I get rid of them, I regain about 200M of space.

I am using the --prune-empty-dirs switch, but it seems to be ignored.

Is this likely the same bug, or different?

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #3 from David <dmorgan at dmorgan.org> 2014-08-10 12:25:58 UTC
---

I wonder if that commit was trying to fix another bug that exists in at least
3.1.0, and possibly other versions.  When an empty directory is deleted from
source but exists in target, and the --delete option is enabled, the empty
directory is not placed in --backup-dir.  Going to see if I can figure out what
exactly the commit changed.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #4 from David <dmorgan at dmorgan.org> 2014-08-10 12:35:52 UTC
---
(In reply to comment #3)
> I wonder if that commit was trying to fix another bug that exists in at
least
> 3.1.0, and possibly other versions.  When an empty directory is deleted
from
> source but exists in target, and the --delete option is enabled, the empty
> directory is not placed in --backup-dir.  Going to see if I can figure out
what
> exactly the commit changed.
Sorry for the self-reply, but I just verified that the bug I noticed exists in
3.0.9, so the commit 3696674bc62b0c1250027dbeedafdd7ebafdcf8b is not related. 
Looks like this needs a new bug report.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |adsh at univ.kiev.ua

--- Comment #5 from Wayne Davison <wayned at samba.org> ---
*** Bug 11423 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #6 from Wayne Davison <wayned at samba.org> ---
In rsync 3.1.x the backup logic was improved to ensure that older, conflicting
files/symlinks couldn't cause problems in a new backup.  For instance, if a
file changed to a directory (and was backed up) and then later a file needed to
be backed up in the new path, the lingering file that now needs to be a
directory caused rsync to die.  The new logic is to ensure that every dir that
exists in the transfer is now a directory in the backup area.  This logic could
be amended to instead ensure that any conflicts are removed, and the dir
creation delayed until needed.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED
                 OS|Mac OS X                    |All

--- Comment #7 from Wayne Davison <wayned at samba.org> ---
This is now fixed in the git source, and will be in the next release.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #8 from adsh <adsh at univ.kiev.ua> ---
Bug is NOT fixed:

rsync --version (on host and client machines)
rsync  version 3.1.2pre1  protocol version 31
...

rsync -btrhv -e "ssh -i /root/.ssh/id_ecdsa_rsync_server"
--delete-after \
--backup-dir=/var/backups/server.ua/pkg/old/`date +%F` \
root at server.ua:/home/user/soft/packages/ /var/backups/server.ua/pkg/data

receiving file list ... done
Created backup_dir /var/backups/server.ua/pkg/old/2015-08-13/

sent 19 bytes  received 7.06K bytes  4.72K bytes/sec
total size is 188.73M  speedup is 26,664.10

ls -l /var/backups/server.ua/pkg/old/2015-08-13/
total 0

---

rsync --version (on host and client machines)
rsync  version 3.0.9  protocol version 30
...

rmdir /var/backups/server.ua/pkg/old/2015-08-13/

rsync -btrhv -e "ssh -i /root/.ssh/id_ecdsa_rsync_server"
--delete-after \
--backup-dir=/var/backups/server.ua/pkg/old/`date +%F` \
root at server.ua:/home/user/soft/packages/ /var/backups/server.ua/pkg/data

receiving file list ... done

sent 11 bytes  received 7.06K bytes  14.13K bytes/sec
total size is 188.73M  speedup is 26705.61

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #9 from adsh <adsh at univ.kiev.ua> ---
Please read https://bugzilla.samba.org/show_bug.cgi?id=10724 carefully.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #10 from adsh <adsh at univ.kiev.ua> ---
Sorry - read this link: https://bugzilla.samba.org/show_bug.cgi?id=11423

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #11 from Wayne Davison <wayned at samba.org> ---
Yes, the top-level backup dir is created at the start of the transfer.  I
don't
plan to change that.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10724

--- Comment #12 from chdiza at gmail.com ---
I can confirm that on rsync 3.1.2pre, the reproduction example I gave at the
start of this thread no longer replicates the issue.  That is, 3.1.2pre appears
at first blush to have fixed the problem.

Thanks!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.