rsync - Apr 2008 - DO NOT REPLY [Bug 5403] New: -H (--hard-links) is broken when sending to remote

https://bugzilla.samba.org/show_bug.cgi?id=5403

           Summary: -H (--hard-links) is broken when sending to remote
           Product: rsync
           Version: 3.0.2
          Platform: x86
        OS/Version: NetBSD
            Status: NEW
          Severity: normal
          Priority: P3
         Component: core
        AssignedTo: wayned@samba.org
        ReportedBy: gcw@pobox.com
         QAContact: rsync-qa@samba.org


Running local to local, hard-link support has exhibited no obvious problems.

Running local to remote, a file hard-linked, say 153 times, would be sent 153
times and stored on the remote 153 times.  Both ends running 3.0.2.
This seems to be independent of recursion options - exhibited without -r, with
-r, and with -r --no-i-r

Not tested:
- interaction between 3.* and 2.*
- running remote to remote, or remote to local


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #1 from gcw@pobox.com  2008-04-21 23:17 CST -------
Running local 2.6.9 to remote 3.0.2 works.

Running local 3.0.2 to remote 3.0.2 works for _some_ hardlinks.  I haven't
isolated which conditions cause failure.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #2 from wayned@samba.org  2008-07-28 20:11 CST -------
Any more info on this?


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #3 from gcw@pobox.com  2008-07-28 20:22 CST -------
Created an attachment (id=3431)
 --> (https://bugzilla.samba.org/attachment.cgi?id=3431&action=view)
output from 2.6.9 to 3.0.3 showing hardlink success


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #4 from gcw@pobox.com  2008-07-28 20:23 CST -------
Created an attachment (id=3432)
 --> (https://bugzilla.samba.org/attachment.cgi?id=3432&action=view)
output from 3.0.3 to 3.0.3 showing hardlink failure (e.g. no => lines)


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED




------- Comment #5 from wayned@samba.org  2008-07-28 20:57 CST -------
Things to try:

Use --protocol=29 on the 3.0.3 -> 3.0.3 transfer and see if that makes the
hard
links work.

Use 3.1.0dev on both systems (either from the git repository or the latest
nightly tar file) and use the --debug=hlink4 option (with no need for so much
general verbosity, and no --protoocl=29 option either) and that might help to
illuminate what is happening.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #6 from gcw@pobox.com  2008-07-28 21:20 CST -------
--protocol=29 with 3.0.3 -> 3.0.3 didn't work

Using --debug=hlink4 with rsync-HEAD-20080727-2332GMT both ends didn't work
either.
# rsync -e rsh -avvRHW --debug=hlink4 --delete-after /rescue remotemachine:/
opening connection using: ...
...
/rescue/zgrep
total: matches=0  hash_hits=0  false_alarms=0 data=652232880
deleting in rescue
sent 652320746 bytes  received 2922 bytes  17871881.32 bytes/sec
total size is 652232880  speedup is 1.00
#


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #7 from wayned@samba.org  2008-07-28 21:53 CST -------
Were there some changes for the rsync to update?

I'd like to see the debug output that was generated by the --debug=hlink4
run.
And know what files should have been hard-linked that weren't.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #8 from gcw@pobox.com  2008-07-28 22:26 CST -------
I'm emptying out the destination directory before each run (though there are
the same results leaving one or two files in there).

There's no difference in the output shown with --debug=hlink4 (except the
first
line redisplaying the running options and the stats line at the end)

Am I missing some compile time thing?

Running with --debug=all4 gives no '=>' outputs.  Is that still
supposed to be
occuring near the end of phase 1?  


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #9 from wayned@samba.org  2008-07-29 09:33 CST -------
Created an attachment (id=3438)
 --> (https://bugzilla.samba.org/attachment.cgi?id=3438&action=view)
Adding 1 to the dev number to avoid a 0

Interesting.  I think that can mean only one thing:  you have a device that has
a number of 0.  Try running this perl command after you cd into the source dir:

perl -e 'print "dev: ", (stat("."))[0],
"\n"'

If it prints "dev: 0", that is the problem and this patch should solve
it.


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403





------- Comment #10 from gcw@pobox.com  2008-07-29 17:20 CST -------
Fantastic.  That is indeed the situation.  "/" mount on NetBSD seems
to always
have device as 0 (at least on all my machines).  The patch did work for me.

I do have a device 0x80000002.  I guess it's conceivable that someone could
have a device 0xffffffff with 32bit device numbers - seems pretty unlikely
though.  If it's only used internally I'm wondering why bother
overloading it.

Still, I'm quite happy with it :-)

Thank you,
Geoff


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

https://bugzilla.samba.org/show_bug.cgi?id=5403


wayned@samba.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED
            Version|3.0.2                       |3.0.4




------- Comment #11 from wayned@samba.org  2008-07-29 20:21 CST -------
I have also added a cast to int64 (which is the internal type used for all
device and inode numbers) so that a 32-bit device number with all bits on will
still be non-zero in the hard-link processing (as long as a dev_t is unsigned,
which it should be).  Of course, a 64-bit device number with all bits on will
overflow to zero, but the current hash code must reserve one number (out of the
18,446,744,073,709,551,615 values available) to indicate that a hash position
is empty, and so choosing an st_dev of 0xffff_ffff_ffff_ffff as the odd-man-out
seems like the best choice.

This fix will go out in the next 3.0.4 pre-release (which will probably be the
final pre-release for 3.0.4).


-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.