rsync - Sep 2010 - DO NOT REPLY [Bug 7670] New: rsync --hard-links fails where ditto succeeds

https://bugzilla.samba.org/show_bug.cgi?id=7670

           Summary: rsync --hard-links fails where ditto succeeds
           Product: rsync
           Version: 3.1.0
          Platform: Other
        OS/Version: Mac OS X
            Status: NEW
          Severity: blocker
          Priority: P3
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: Dave at Yost.com
         QAContact: rsync-qa at samba.org


Given enough hard links in a source folder and a small enough destination
volume, copying with rsync -- hard-links can fail whereas the Mac OS X ditto
program succeeds. Rsync fails by exhausting the maximum number of hard links on
the destination.

I have made a script that demonstrates this bug. It makes two minimal ram
disks, with a volume on each. On the source volume it makes a maximal set of
hard linked files that can be copied correctly to the other volume using ditto
but not using rsync.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #1 from Dave at Yost.com  2010-09-09 19:40 CST -------
Created an attachment (id=5956)
 --> (https://bugzilla.samba.org/attachment.cgi?id=5956&action=view)
test script that demonstrates the bug


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #2 from Dave at Yost.com  2010-09-09 19:42 CST -------
test script output looks like this

rsync: mkstemp "/Volumes/Copy/links/03/29/.83b.QlmOQ8" failed: No
space left on
device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.84b.KFW6da" failed: No
space left on
device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.85b.vB6pC5" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/36a" => links/03/28/36b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.86b.jBXuVJ" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/37a" => links/03/28/37b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.87b.C3svVR" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/38a" => links/03/28/38b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.88b.RcRxZH" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/39a" => links/03/28/39b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.89b.mFP87z" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/40a" => links/03/28/40b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.90b.GQbvqB" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/41a" => links/03/28/41b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.91b.L5bJ8n" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/42a" => links/03/28/42b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.92b.PXp0X0" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/43a" => links/03/28/43b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.93b.vpOJOO" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/44a" => links/03/28/44b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.94b.8Y8sFs" failed: No
space left on
device (28)
rsync: link "/Volumes/Copy/links/03/28/45a" => links/03/28/45b
failed: No space
left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.95b.41bN0l" failed: No
space left on
device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.96b.rMQnZB" failed: No
space left on
device (28)
...


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #3 from c182driver29 at gideon.org  2010-09-14 08:33 CST -------
Is this really an rsync bug?  I experienced the same thing on Linux with ext3
with our backup system, which is heavily dependent upon --link-dest.  I've
tested switching to xfs, and this solves the problem because the real issue
isn't rsync but instead the underlying file system.

I also confirmed that cp -Rl had the same problem and solution.

http://www.mail-archive.com/rsync at lists.samba.org/msg25950.html

I did wonder whether rsync should, when faced with this error, fall back on
creating a copy.  But should rsync include behavior that exists only to work
around a file system limit?  Perhaps only as a command line option (ie.
definitely not the default behavior)?

I'm also not sure that it should always do this.  It's not so bad to do
so for
--link-dest, which is really just an optimization.  But if rsync is to be
making a reliable copy of the source at the destination, this should include
mimicry of any links.  In that case, falling back to copy would be a bad idea,
I believe.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #4 from Dave at Yost.com  2010-09-14 12:47 CST -------
On Mac OS X, the ditto program works where rsync fails, so it seems this is a
problem with rsync (and now we know it is also a problem with "cp
-Rl"). Wayne,
perhaps you could ask a friend at Apple how ditto does the equivalent of
--hard-links.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #5 from Dave at Yost.com  2011-01-27 18:37 CST -------
Why is this not a blocker? Why is it still marked New?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #6 from matt at mattmccutchen.net  2011-01-27 18:48 CST -------
(In reply to comment #5)
> Why is this not a blocker?
It does not "block development and/or testing work"
(https://bugzilla.samba.org/page.cgi?id=fields.html#bug_severity).  An example
of a blocker would be if rsync were unable to add directories to the file list
at all; then very little meaningful testing could be done.
> Why is it still marked New?
No one has started to work on it.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #7 from samba at jonix.dk  2011-02-18 06:38 CST -------
I experience the same problem on Linux rsync'ing over the network. Both
hosts
uses XFS filesystem(s), both uses Volumegroups that has been lvextended and
then xfs_growfs.

Source uses several smaller LV in different VG (some SSD, some HDD), all XFS
and mounted in various locations.

Destination uses 1 big LV in 1 VG.

rsync source: tried both debian lenny and squeeze packages, 3.0.3-2 and
3.0.7-2.
rsync destination debian lenny package, 3.0.3-2

Kernel source 2.6.32-5-vserver-amd64
kernel destination 2.6.22.18-vs2.2.0.6

Hardware source: Quad-Core AMD Opteron(tm) Processor 2378
hardware destination: Intel(R) Pentium(R) 4 CPU 3.20GHz

df -hi reports plenty of free inodes. Source uses from 1-2%, destination 1%.
There is also several GB left.

Setup: Source is the new server, destination was the old server which is now
redelegated to hotspare (soon to be offsite) to avoid having to restore from
backup if something bad happens.

Will try upgrading destination to squeeze. Upgrading did not work, same result.

rsync command running as root rsync -a -v -P --stats --numeric-ids
--exclude=/dev --exclude=/proc --exclude=/sys root@<servername>:/* .


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670


samba at jonix.dk changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |samba at jonix.dk




------- Comment #8 from samba at jonix.dk  2011-02-18 07:46 CST -------
Some old unchanged files are transfered again, the sysadm/Guide ...
files and directories are not transfered?

data/cvs/sysadm/Guide to Partner access a.doc,v
     1149519 100%   88.75MB/s    0:00:00 (xfer#4046, to-check=1009/316808)
data/cvs/sysadm/Active-Directory-sync/
rsync: recv_generator: mkdir
"/mnt/data/cvs/sysadm/Active-Directory-sync"
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
data/cvs/sysadm/cvsmail/
rsync: recv_generator: mkdir "/mnt/data/cvs/sysadm/cvsmail" failed: No
space
left on device (28)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/mnt/data/cvs/sysadm/.Guide to Partner access
a.doc,v.UpUy7A"
failed: No space left on device (28)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #9 from samba at jonix.dk  2011-02-18 08:14 CST -------
Running xfs_info on my destination filesystem gives me:

meta-data=/dev/mapper/VG-<name>   isize=256    agcount=5, agsize=91750400
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=419430400, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=7670





------- Comment #10 from samba at jonix.dk  2011-02-21 04:54 CST -------
I reformated my LV and transfered the files again over the weekend. It works
fine. Even when I rerun rsync monday morning. It also works after extending the
XFS filesystem. The difference could be that this time the filesystem did not
run completely out of space during the rsync. It does not matter if I push or
pull using rsync.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.