rsync - Feb 2016 - --link-dest not working on remote server (running daemon)

Hello,

I have read through the list of previous issues regarding this issue but
haven't been able to resolve mine. I apologize in advance for the long text
and am probably doing some simple typo. I have two servers in my setup:

*Server 1*
Doing rsync with --link-dest daily and working as expected. I'm getting the
hard links in the new daily directories.

*Server 2*
Running rsync daemon mode with following config

[offsite]
   path = /media/external/backup/
   comment = Offsite backup
   read only = no
   hosts allow = 192.168.2.0/24
   auth users = backup
   secrets file = /etc/rsyncd.scrt
   uid = 0
   gid = 0


*My Goal*
Server 1 has been running for several months now so it has several months
of daily backups. I was able to do an initial sync to server 2 using -H
option to keep my hardlink structure. That worked fine and my original plan
was to just run the same rsync with -H after the daily backup was complete
to keep both in sync. But that turned out to be very slow building the
incremental file list as I'm guessing it scanned all files for each daily
backup (even though they were hard lnked).

So my next plan was to just sync that latest daily backup from server 1 to
server 2 and use the --link-dest option on server 2 to link it to the
previous day.

*The Problem*
This is the command I'm troubleshooting right now

rsync -a -v -n -i --delete --link-dest=/backup-2016-02-01-0100
--password-file=/media/external/scripts/offsite_rsync.pass
/media/external/backup/backup-2016-02-02-0100 backup at 192.168.2.102::offsite

It seems to be sending all files as new files (i.e. not picking up the
link-dest option). I've tried using no slash at the beginning of link-dest,
tried using ./ tried full path. etc.

Here is a snippet of the output from server 1 which is running the command:

<f+++++++++
backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120
DC Air and Space Museum/IMGP0906.JPG


On server 2 there is no backup-2016-02-02-0100 directory. However, the
link-dest option I'm using has the file on server 2:

-rw------- 147 www-data www-data 4454193 Jan 20  2007
/media/external/backup/backup-2016-02-01-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120
DC Air and Space Museum/IMGP0906.JPG

Output of same file from server 1

-rw------- 151 www-data www-data 4454193 Jan 20  2007
/media/external/backup/backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2007/20070120
DC Air and Space Museum/IMGP0906.JPG


*Troubleshooting*
If I manually create the copy of server 2 first (cp -al
backup-2016-02-01-0100 backup-2016-02-02-0100) and then run rsync without
the --link-dest option I get the expected results. Only the files that
changed or were removed/added are transferred.

Also I tried running the actual transfer without -n and it is indeed
transferring all the old files.

Any help is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20160208/b9d6c789/attachment.html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/08/2016 03:33 PM, Sam Holton wrote:
> Hello,
> 
> I have read through the list of previous issues regarding this
> issue but haven't been able to resolve mine. I apologize in advance
> for the long text and am probably doing some simple typo. I have
> two servers in my setup:
> 
> *Server 1* Doing rsync with --link-dest daily and working as
> expected. I'm getting the hard links in the new daily directories.
> 
> 
> *Server 2* Running rsync daemon mode with following config
> 
> [offsite] path = /media/external/backup/ comment = Offsite backup 
> read only = no hosts allow = 192.168.2.0/24
> <http://192.168.2.0/24> auth users = backup secrets file >
/etc/rsyncd.scrt uid = 0 gid = 0
Why are you using rsyncd instead of rsync over ssh?  Is --link-dest
even supposed to be supported via rsyncd?  Even if it is, you seem to
have your link-dest and your target parameters misaligned.
> 
> 
> *My Goal* Server 1 has been running for several months now so it
> has several months of daily backups. I was able to do an initial
> sync to server 2 using -H option to keep my hardlink structure.
> That worked fine and my original plan was to just run the same
> rsync with -H after the daily backup was complete to keep both in
> sync. But that turned out to be very slow building the incremental
> file list as I'm guessing it scanned all files for each daily
> backup (even though they were hard lnked).
It has to scan them to know that.
> So my next plan was to just sync that latest daily backup from
> server 1 to server 2 and use the --link-dest option on server 2 to
> link it to the previous day.
> 
> *The Problem* This is the command I'm troubleshooting right now
> 
> rsync -a -v -n -i --delete --link-dest=/backup-2016-02-01-0100 
> --password-file=/media/external/scripts/offsite_rsync.pass 
> /media/external/backup/backup-2016-02-02-0100
> backup at 192.168.2.102::offsite
> 
> It seems to be sending all files as new files (i.e. not picking up
> the link-dest option). I've tried using no slash at the beginning
> of link-dest, tried using ./ tried full path. etc.
> 
> Here is a snippet of the output from server 1 which is running the
> command:
> 
> <f+++++++++ 
> backup-2016-02-02-0100/media/external/owncloud/data/sam/files/Photos/2
007/20070120
>
> 
DC Air and Space Museum/IMGP0906.JPG
> 
> 
> On server 2 there is no backup-2016-02-02-0100 directory. However,
> the link-dest option I'm using has the file on server 2:
> 
> -rw------- 147 www-data www-data 4454193 Jan 20  2007 
> /media/external/backup/backup-2016-02-01-0100/media/external/owncloud/
data/sam/files/Photos/2007/20070120
>
> 
DC Air and Space Museum/IMGP0906.JPG
> 
> Output of same file from server 1
> 
> -rw------- 151 www-data www-data 4454193 Jan 20  2007 
> /media/external/backup/backup-2016-02-02-0100/media/external/owncloud/
data/sam/files/Photos/2007/20070120
>
> 
DC Air and Space Museum/IMGP0906.JPG
> 
> 
> *Troubleshooting* If I manually create the copy of server 2 first
> (cp -al backup-2016-02-01-0100 backup-2016-02-02-0100) and then run
> rsync without the --link-dest option I get the expected results.
> Only the files that changed or were removed/added are transferred.
> 
> Also I tried running the actual transfer without -n and it is
> indeed transferring all the old files.
> 
> Any help is appreciated.
> 
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAla4/lUACgkQVKC1jlbQAQcLsACfeC7yjw+ooxoKbYlupzcRYOb2
RiQAoN83Cy9t/7UMExoz1kG49556O6fS
=GPzH
-----END PGP SIGNATURE-----