rsync - Apr 2012 - [Bug 8856] New: --hard-links does not handle hard-linked symlinks correctly on FreeBSD

https://bugzilla.samba.org/show_bug.cgi?id=8856

           Summary: --hard-links does not handle hard-linked symlinks
                    correctly on FreeBSD
           Product: rsync
           Version: 3.0.7
          Platform: All
        OS/Version: FreeBSD
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: ronaldj at infoxchange.net.au
         QAContact: rsync-qa at samba.org


Creating hard-linked symlinks fails on FreeBSD RELEASE-8.2.

Simple test case:

# uname -sr
FreeBSD 8.2-RELEASE
# ls -liR
total 4
117761 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 a
117762 drwxr-xr-x  2 root  wheel  512 Apr 11 11:42 b

./a:
total 0
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z

./b:
total 0
# rsync -avH a/ b
sending incremental file list
./
lf
s1 -> f
rsync: link "/root/test/b/ls1" => s1 failed: No such file or
directory (2)
s2 -> /x/y/z
rsync: link "/root/test/b/ls2" => s2 failed: No such file or
directory (2)
f => lf

sent 155 bytes  received 50 bytes  410.00 bytes/sec
total size is 14  speedup is 0.07
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1042) [sender=3.0.7]
# ls -liR
total 4
117761 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 a
117762 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 b

./a:
total 0
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z

./b:
total 0
117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117766 lrwxr-xr-x  1 root  wheel  1 Apr 11 10:30 s1 -> f
117767 lrwxr-xr-x  1 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z
# rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, no iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.


So it not only doesn't hard-link the symlinks in the target directory, it
does
not even create any in the first place.


Following the exact same steps on Debian squeeze for verification:

# uname -sr
Linux 2.6.32-5-686-bigmem
# ls -liR
.:
total 8
193954 drwxr-xr-x 2 root root 4096 Apr 11 10:50 a
193955 drwxr-xr-x 2 root root 4096 Apr 11 11:49 b

./a:
total 0
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z

./b:
total 0
# rsync -avH a/ b
sending incremental file list
./
lf
s1 -> f
ls1 => s1
s2 -> /x/y/z
ls2 => s2
f => lf

sent 175 bytes  received 70 bytes  490.00 bytes/sec
total size is 14  speedup is 0.06
# ls -liR
.:
total 8
193954 drwxr-xr-x 2 root root 4096 Apr 11 10:50 a
193955 drwxr-xr-x 2 root root 4096 Apr 11 10:50 b

./a:
total 0
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z

./b:
total 0
193961 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193961 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193959 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193960 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193959 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193960 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z
# rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.



Apologies if this has already been fixed or/and reported - I did search
Bugzilla but could not find anything.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

There is no such thing as a hard link of a symlink.  A "hard link" is
nothing more than an additional filename referencing an existing
inode.  A symlink is pointer to a link.  There is no such thing as a
filename that references a pointer to itself.

On 04/10/12 21:55, samba-bugs at samba.org wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=8856
> 
> Summary: --hard-links does not handle hard-linked symlinks 
> correctly on FreeBSD Product: rsync Version: 3.0.7 Platform: All 
> OS/Version: FreeBSD Status: NEW Severity: normal Priority: P5 
> Component: core AssignedTo: wayned at samba.org ReportedBy:
> ronaldj at infoxchange.net.au QAContact: rsync-qa at samba.org
> 
> 
> Creating hard-linked symlinks fails on FreeBSD RELEASE-8.2.
> 
> Simple test case:
> 
> # uname -sr FreeBSD 8.2-RELEASE # ls -liR total 4 117761 drwxr-xr-x
> 2 root  wheel  512 Apr 11 10:31 a 117762 drwxr-xr-x  2 root  wheel
> 512 Apr 11 11:42 b
> 
> ./a: total 0 117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f 
> 117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf 117764
> lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f 117765
> lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z 117764
> lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f 117765 lrwxr-xr-x
> 2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z
> 
> ./b: total 0 # rsync -avH a/ b sending incremental file list ./ lf 
> s1 -> f rsync: link "/root/test/b/ls1" => s1 failed: No
such file
> or directory (2) s2 -> /x/y/z rsync: link "/root/test/b/ls2"
=> s2
> failed: No such file or directory (2) f => lf
> 
> sent 155 bytes  received 50 bytes  410.00 bytes/sec total size is
> 14  speedup is 0.07 rsync error: some files/attrs were not
> transferred (see previous errors) (code 23) at main.c(1042)
> [sender=3.0.7] # ls -liR total 4 117761 drwxr-xr-x  2 root  wheel
> 512 Apr 11 10:31 a 117762 drwxr-xr-x  2 root  wheel  512 Apr 11
> 10:31 b
> 
> ./a: total 0 117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f 
> 117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf 117764
> lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f 117765
> lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z 117764
> lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f 117765 lrwxr-xr-x
> 2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z
> 
> ./b: total 0 117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f 
> 117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf 117766
> lrwxr-xr-x  1 root  wheel  1 Apr 11 10:30 s1 -> f 117767 lrwxr-xr-x
> 1 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z # rsync --version rsync
> version 3.0.7  protocol version 30 Copyright (C) 1996-2009 by
> Andrew Tridgell, Wayne Davison, and others. Web site:
> http://rsync.samba.org/ Capabilities: 64-bit files, 32-bit inums,
> 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks,
> symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, no
> iconv, symtimes
> 
> rsync comes with ABSOLUTELY NO WARRANTY.  This is free software,
> and you are welcome to redistribute it under certain conditions.
> See the GNU General Public Licence for details.
> 
> 
> So it not only doesn't hard-link the symlinks in the target
> directory, it does not even create any in the first place.
> 
> 
> Following the exact same steps on Debian squeeze for verification:
> 
> # uname -sr Linux 2.6.32-5-686-bigmem # ls -liR .: total 8 193954
> drwxr-xr-x 2 root root 4096 Apr 11 10:50 a 193955 drwxr-xr-x 2 root
> root 4096 Apr 11 11:49 b
> 
> ./a: total 0 193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f 193956
> -rw-r--r-- 2 root root 0 Apr 11 10:49 lf 193957 lrwxrwxrwx 2 root
> root 1 Apr 11 10:49 ls1 -> f 193958 lrwxrwxrwx 2 root root 6 Apr 11
> 10:49 ls2 -> /x/y/z 193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1
> -> f 193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z
> 
> ./b: total 0 # rsync -avH a/ b sending incremental file list ./ lf 
> s1 -> f ls1 => s1 s2 -> /x/y/z ls2 => s2 f => lf
> 
> sent 175 bytes  received 70 bytes  490.00 bytes/sec total size is
> 14  speedup is 0.06 # ls -liR .: total 8 193954 drwxr-xr-x 2 root
> root 4096 Apr 11 10:50 a 193955 drwxr-xr-x 2 root root 4096 Apr 11
> 10:50 b
> 
> ./a: total 0 193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f 193956
> -rw-r--r-- 2 root root 0 Apr 11 10:49 lf 193957 lrwxrwxrwx 2 root
> root 1 Apr 11 10:49 ls1 -> f 193958 lrwxrwxrwx 2 root root 6 Apr 11
> 10:49 ls2 -> /x/y/z 193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1
> -> f 193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z
> 
> ./b: total 0 193961 -rw-r--r-- 2 root root 0 Apr 11 10:49 f 193961
> -rw-r--r-- 2 root root 0 Apr 11 10:49 lf 193959 lrwxrwxrwx 2 root
> root 1 Apr 11 10:49 ls1 -> f 193960 lrwxrwxrwx 2 root root 6 Apr 11
> 10:49 ls2 -> /x/y/z 193959 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1
> -> f 193960 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z #
> rsync --version rsync  version 3.0.7  protocol version 30 Copyright
> (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web
> site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit
> inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks,
> symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv,
> symtimes
> 
> rsync comes with ABSOLUTELY NO WARRANTY.  This is free software,
> and you are welcome to redistribute it under certain conditions.
> See the GNU General Public Licence for details.
> 
> 
> 
> Apologies if this has already been fixed or/and reported - I did
> search Bugzilla but could not find anything.
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk+E5lkACgkQVKC1jlbQAQft6wCfZwvUJL1gAcSRINYRCJh1iWIL
/m8AoMv8P5V0J9WbadcQP6jAU5GQbIqM
=0K7Z
-----END PGP SIGNATURE-----

What are the FreeBSD and Linux 'a' filesystem types?
Please post the list of commands used to recreate
the FreeBSD directory 'a', starting from rm -rf ./a.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #1 from ron <ronaldj at infoxchange.net.au> 2012-04-11
04:05:36 UTC ---
> What are the FreeBSD and Linux 'a' filesystem types?
FreeBSD: ufs
Linux: ext3
>Please post the list of commands used to create the FreeBSD directory
'a', starting from rm -rf ./a.
# rm -rf ./a
# mkdir a
# cd a
# touch f
# ln f lf
# ln -s f s1
# ln -s /x/y/z s2
# ln -P s1 ls1
# ln -P s2 ls2

I created two symbolic links to check whether the issue has anything to do with
the target existing (as with f) or not (as with /x/y/z).

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #2 from grarpamp at gmail.com 2012-04-11 04:51:57 UTC ---
Ok, somehow I missed -P, thanks :)

I had meant to exhaustively test all these various
things with -Ha (and optionally --link-dest) back
when I was reporting those sorts of bugs but never
got around to it. Someone really should work up a
make test for it all.

[1] Inode consumers
.
..
dir
file
char
block
fifo
socket
slink

[2] Name entries
hlink (all but dirs are hard linkable)

[3] Meta info
perm
user/group
birth/modify/access/change time

FreeBSD is moving its syscalls towards being able to
change any of the properties in [3] for entities
in [1], there are some left to go. It's not a bad
or nonsensical thing either, just an inode.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #3 from ron <ronaldj at infoxchange.net.au> 2012-04-12
00:27:12 UTC ---
It sure makes sense to handle all those entities in the same way, even though
it gets confusing when the same command acts on them in different ways. As in
the case of hard-linking symlinks, where the default is to hard-link the
symlink target instead of the symlink itself.

Pretty big (t)ask to verify all permutations of configurations, command line
options and operating systems, in any case. :-)

Thanks for looking into this, it's much appreciated.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #4 from Wayne Davison <wayned at samba.org> 2013-05-27
00:18:04 UTC ---
Look at the test in configure.ac where it checks if link() can hard-link
symlinks and see what needs to change for FreeBSD.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #5 from awk <sambabugzilla201308 at awk.bz> 2013-08-07
06:55:08 UTC ---
Same behavior on:

FreeBSD 9.x ZFS (and UFS)
OmniOS (OpenSolaris) ZFS

In fact it often just hangs outright after the "No such file or
directory"
error message.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #6 from Wayne Davison <wayned at samba.org> 2013-08-08
16:02:13 UTC ---
You should note that this bug is awaiting info that nobody has bothered to
provide.  Nothing more is going to be done until someone figures out what is
needed.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #7 from awk <sambabugzilla201308 at awk.bz> 2013-08-08
17:44:36 UTC ---
Well, under FreeBSD linkat() will hardlink symlinks, i.e. instead of:

link(source, target);

use:

linkat(AT_FDCWD, source, AT_FDCWD, target, 0);

I patched my copy of rsync to use that and it worked when copying files, but
not when using --dry-run (still printed errors / hung).

So that's part of it, at least. I plan on digging into it further when I
have a
chance.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #8 from sylvain at ilm-informatique.fr ---
To expand on the previous post : POSIX previously mandated that link() resolve
the target (as FreeBSD does), but some systems (including Linux) did not. So in
the last standard linkat() was added (with a new parameter) to disambiguate. So
if linkat() is available it should be used, it's available at least on
FreeBSD
>= 8.0 (released 23 July 2010) and on Linux >= 2.6.18 (released 20
September
2006) and glibc >= 2.4.

Sources: 
http://pubs.opengroup.org/onlinepubs/9699919799/functions/linkat.html
http://manpages.ubuntu.com/manpages/trusty/man2/link.2.html
http://manpages.ubuntu.com/manpages/trusty/man2/linkat.2.html
https://www.freebsd.org/cgi/man.cgi?query=link&sektion=2&manpath=FreeBSD+10.1-RELEASE+and+Ports&arch=default&format=html

I've patched configure.ac and syscall.c of rsync 3.1.1 to use linkat() on my
FreeBSD 9.1 system and I haven't have any problems (including with dry run,
contrary to the previous post).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #9 from sylvain at ilm-informatique.fr ---
Created attachment 10435
  --> https://bugzilla.samba.org/attachment.cgi?id=10435&action=edit
configure.ac patch

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=8856

--- Comment #10 from sylvain at ilm-informatique.fr ---
Created attachment 10436
  --> https://bugzilla.samba.org/attachment.cgi?id=10436&action=edit
syscall.c patch

A #ifdef HAVE_LINKAT should probably be added

-- 
You are receiving this mail because:
You are the QA Contact for the bug.