rsync - Jan 2016 - [Bug 11521] rsync does not use high-resolution timestamps to determine file differences

On systems using nanoseconds differences should be taken into consideration.

--- a/generator.c	2014-06-14 01:05:08.000000000 +0200
+++ b/generator.c	2014-12-25 11:19:54.000000000 +0100
@@ -588,7 +588,13 @@
 	if (ignore_times)
 		return 0;

-	return cmp_time(st->st_mtime, file->modtime) == 0;
+	return cmp_time(st->st_mtime, file->modtime) == 0
+#ifdef ST_MTIME_NSEC
+	       ? NSEC_BUMP(file) ? (uint32)st->ST_MTIME_NSEC == F_MOD_NSEC(file)
+	                         : 1
+	       : 0
+#endif
+	;
 }

Hi,

I obviously didn't think of --modify-window, so in order to not behave
erratically it should be at least:

--- a/generator.c	2014-06-14 01:05:08.000000000 +0200
+++ b/generator.c	2015-01-02 15:50:30.000000000 +0100
@@ -588,7 +588,14 @@
 	if (ignore_times)
 		return 0;

-	return cmp_time(st->st_mtime, file->modtime) == 0;
+	return cmp_time(st->st_mtime, file->modtime) == 0
+#ifdef ST_MTIME_NSEC
+	       ? st->st_mtime == file->modtime
+	         && NSEC_BUMP(file) ? (uint32)st->ST_MTIME_NSEC ==
F_MOD_NSEC(file)
+	                            : 1
+	       : 0
+#endif
+	;
 }



Most probably, the check should be part of cmp_time(), but I can't overview
possible consequences and I don't know whether such a change would have a
chance to be accepted and hence is worth the effort.

Ingo

I wrote on Fri, 02 Jan 2015 16:02:27 +0100:
> --- a/generator.c       2014-06-14 01:05:08.000000000 +0200
> +++ b/generator.c       2015-01-02 15:50:30.000000000 +0100
> @@ -588,7 +588,14 @@
>         if (ignore_times)
>                 return 0;
> -       return cmp_time(st->st_mtime, file->modtime) == 0;
> +       return cmp_time(st->st_mtime, file->modtime) == 0
> +#ifdef ST_MTIME_NSEC
> +              ? st->st_mtime == file->modtime
> +                && NSEC_BUMP(file) ? (uint32)st->ST_MTIME_NSEC
=> F_MOD_NSEC(file)
> +                                   : 1
> +              : 0
> +#endif
> +       ;
>  }
Ping?

Unfortunately, there weren't any comments yet.

Ingo

On Thu, Dec 25, 2014 at 2:48 AM, Ingo Brückl <ib at wupperonline.de>
wrote:
> On systems using nanoseconds differences should be taken into
> consideration.
>
The problem is that if you transfer from a filesystem that has nanoseconds
to one that does not support it, rsync would consider most of the files to
be constantly different, since the nanosecond values would only match if
the source file happened to have 0 nanoseconds. So, the logic has to be
improved to somehow detect such a case and treat the truncated values as
equal. One possible improvement would be to skip the nanosecond check if
the destination file has a nanosecond value of 0.  That could possibly be
improved if we figure out if a particular device ID supports nanoseconds
somehow.  I have a potential heuristic in mind that I can code up and see
how it works.

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20160120/3af364c9/attachment.html>

On Wed 20 Jan 2016, Wayne Davison wrote:
> equal. One possible improvement would be to skip the nanosecond check if
> the destination file has a nanosecond value of 0.  That could possibly be
> improved if we figure out if a particular device ID supports nanoseconds
> somehow.  I have a potential heuristic in mind that I can code up and see
> how it works.
It would be very handy if that could be extended to take into account
FAT filesystems, i.e. automagically set --modify-window=2 if it's
detected that you're writing to a FAT filesystem. It's basically the
same problem as detecting that you're writing to a filesystem that
doesn't support nanoseconds IMHO.


Paul

> Date: Wed, 20 Jan 2016 23:04:20 -0800
    > From: Wayne Davison <wayned at samba.org>

    > On Thu, Dec 25, 2014 at 2:48 AM, Ingo Br=C3=BCckl <ib at
wupperonline.de> wrote:

    > > On systems using nanoseconds differences should be taken into
    > > consideration.

    > The problem is that if you transfer from a filesystem that has
nanoseconds
    > to one that does not support it, rsync would consider most of the files
to
    > be constantly different, since the nanosecond values would only match
if
    > the source file happened to have 0 nanoseconds. So, the logic has to be
    > improved to somehow detect such a case and treat the truncated values
as
    > equal. One possible improvement would be to skip the nanosecond check
if
    > the destination file has a nanosecond value of 0.  That could possibly
be
    > improved if we figure out if a particular device ID supports
nanoseconds
    > somehow.  I have a potential heuristic in mind that I can code up and
see
    > how it works.

Here's one idea, and note an important issue with ns times and --link-dest:

(a) For each end, see if any of the files being considered already
have nonzero nanosecond parts.  If so, then that end of the transfer
supports nanosecond timing.

(b) If the sending filesystem appears to have nonzero ns parts, and
the receiving filesystem appears to have all-zero ns parts (including
any directories under consideration), the receiver may still support
ns times, but have been synchronized from a filesystem that didn't.
We don't want to perpetuate that on the -next- sync, however, so we
can't just disallow ns times on the receiver, or we'll never try them
again.

(c) In case (b) above, therefore, if any file to be transmitted has a
ns time, transfer it and then immediately check the received file's
timestamp.  If its ns time is still zero, then the receiving
filesystem doesn't support it, so disable ns times during the
transfer.  If it's nonzero, then enable.  (I am eliding the pipelining
that happens during an actual rsync; that may have to be dealt with
somehow.)  Also, check the directory mod time, and see if -that's- now
nonzero; you have a very small chance of it being zero if ns times are
supported, and you can check for being in or near that window.  And
the first time it's nonzero in this filesystem, you know it'll work
for everything else in this fs.*

* "This filesystem" assumes either that you can detect mountpoints,
or that the heuristic should be applied per-directory, and that no
directory has a single-file mountpoint that doesn't support it, etc.
I assume rsync must already have some sort of logic like this for
dealing with xattr support per-fs, etc.  If this is flaky to do,
then you might need --[no]ns-timing switches to force rsync to do
the right thing without complaining on every single file if it guesses
wrong.

I don't know if the rsync protocol is flexible enough to dynamically
enable or disable this capability partway through a transfer.  If it
isn't, then there's an even more hackish approach, which is to add,
and unconditionally attempt to honor, a --ns-times-valid sort of
switch.  Users can then use the heuristics above in a dummy transfer
to know whether to set that switch for the real transfer.  (Or they
may know out-of-band that their FS supports ns times.)  But I'd think
such a switch and workaround should be last resorts.

I would really like to see ns times supported.  I use dirvish to back
up filesystems, which uses rsync, and if I ever have to restore any
files from that (which I do more often due to accidentally deleting or
bashing a file than due to media failure), I lose the ns timestamps,
and they're sometimes extremely valuable forensically when I'm trying
to debug something else.  Having them be 0 when I thought they shouldn't
be has more than once cost me time until I realized that I'm looking
at files that were rsync'ed from another host (either to duplicate
a setup, or from a backup) and rsync didn't preserve the ns times.

Unfortunately, of course, if rsync gets fixed now, it -will- consider
every single backed-up file in my dirvish vaults to be "new" and will
insanely bloat the vault (doubling its size) the next time it runs,
and then I'll have to tell faster-dupemerge how to re-merge all that
stuff, too.  (After all, even if the file contents haven't changed,
its metadata has, so --link-dest is required to create a new copy
of the file rather than hardlink to one with a different timestamp.)

What I'd -really- like is for some sane interaction with --link-dest
as well (which probably requires another switch, alas), which
basically says "a change from ns-0 to ns-other with no other changes
to the file is considered the same file---update the timestamp to the
new ns time but don't break the hardlink", with a way of forcing that
off for people who aren't in my situation and do care about such a
change.  Failing that, I'd need to do something like (a) run a backup
in non-ns mode by force, then (b) immediately re-run the backup in
ns-mode -on the same output directories-, e.g., -not- using
--link-dest to create new dirvish vaults.  This should get the times
resynchronized without breaking all the hardlinks to the previous
backups.  (I suspect that this would force ns times into files dozens
of generations back in the vaults, since those hardlinks would all
share metadata, but that's okay and in fact desireable.)

Note that this change in rsync behavior would thus appear to need a
pretty big warning in the changelog and new-version announcements
warning people that those who use --link-dest (which I assume means
by-hand, via dirvish, and via rsnapshot, at least) need to make some
sort of workaround (TBD) so as not to have their backups suddenly
explode in both time and space.  I -still- think I'd like to see
ns times in rsync, despite this caveat---the longer it's delayed, the
worst the situation gets.  (A coordinated change in the most-popular
tools that use --link-dest to implement a workaround or at least
warn the user also seems wise; otherwise, those who upgrade their
OS and get a new version of rsync that way, without reading release
notes, may be surprised.  Which means such tools need a way of knowing
which rsync implements ns times, presumably by adding it to the
"Capabilities" output of --version or something.  Unless, of course,
the ns-0-to-ns-other-means-same-file-for-link-dest is the default,
which I think is what I'd recommend, as long as there's a way to
turn that off and it's well-documented.)

P.S. The current situation also means that faster-dupemerge can't use
that information, either, because I can't trust it to be correct
across hosts in such situations.  [I made a version of f-d that
respected ns times, only to abandon it when I realized that rsync
wasn't preserving them!]  I merge -across- vaults with f-d to catch
files that are the same on multiple backed-up hosts, or to catch
pushing a file from one host to another and deleting it from the
original host, or to merge identical files on same host in the backup
even if they aren't merged on the host being backed up.

[Paul Slootman's request for FAT filesystems would be a generalization
of this sort of strategy, although I'd think that in that case it's a
lot more obvious to the user invoking rsync that the fs is FAT.]

On Wed, 20 Jan 2016 23:04:20 -0800
Wayne Davison <wayned at samba.org> wrote:
> 
> The problem is that if you transfer from a filesystem that has
> nanoseconds to one that does not support it, rsync would consider
> most of the files to be constantly different, since the nanosecond
> values would only match if the source file happened to have 0
> nanoseconds. So, the logic has to be improved to somehow detect such
> a case and treat the truncated values as equal. One possible
> improvement would be to skip the nanosecond check if the destination
> file has a nanosecond value of 0.  That could possibly be improved if
> we figure out if a particular device ID supports nanoseconds
> somehow.
Seems to me that nanoseconds are the sort of thing that could cause
sysadms crazy headaches.   My thought is to have a declaration
in the rsync configuration file (that can be overridden on
the command line).    Something like "--nanosecond".
It'd have the following values:

ignore   : Ignore nanoseconds.  (default)

update   : Ignore nanoseconds, but update destination timestamps
           when nanoseconds differ.

heuristic: Check nanoseconds with Wayne's spiffy heuristic.

check    : Check nanoseconds.

When there is a conflict between the conf files of the 2 endpoints the
topmost of the above options has priority.  (When no configuration
is specified on at least one endpoint there is no conflict.)

To provide control over conflict management you could have another
option, say, --nanosecond-force, to force your endpoint's choice.  If
both ends force then the later of ignore, update, heuristic, check has
priority.

I don't know how this would work with the existing rsync protocol.
Perhaps it'd be easier to have only the destination end's config
matter, although this does not provide a lot of flexibility from the
command line.  The motivation is to be able to keep things simple,
or as simple as they can be.  Already my ideas seem overly complicated.
Perhaps someone can improve them.

It makes some sense to be able to configure nanosecond related
behavior on a per-directory (i.e., mountpoint) basis, as a substitute
for knowing about every possible filesystem type and being able to
detect file system type.  But this introduces yet more complication.

The default is backward compatible.  Distros could set their
own default in the rsync.conf file they install.

---

Maybe the thing to do is to give up on runtime complication and
just do testing on the destination filesystem when rsync initializes.
This would be on by default but could be turned off by command line.

Since the problem is with destination filesystems that don't support
nanoseconds, and destination filesystems are by definition written to,
then test for nanosecond support once when rsync starts.  Write a file
with non-zero nanoseconds and read it back and see if the nanoseconds
are zero.  Then delete the test file.  Easy if the destination is
empty, harder if there's an existing directory hierarchy.  (But rsync
can already tell if it's crossing a filesystem boundary so....)

Trouble is that unconditionally writing a file would affect directory
timestamps.  If you wait until you know you're writing to a directory
then the approach here is starting to sound suspiciously like Wayne's
heuristic....

Hope these thoughts are helpful to somebody.

Regards,

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

[Included text copied from the commit.]

    > Date: Sun, 24 Jan 2016 19:52:49 +0000
    > From: samba-bugs at samba.org

    > Auto-Submitted: auto-generated

    > https://bugzilla.samba.org/show_bug.cgi?id=11521

    > Wayne Davison <wayned at samba.org> changed:

    >            What    |Removed                     |Added
    >
----------------------------------------------------------------------------
    >              Status|NEW                         |RESOLVED
    >          Resolution|---                         |FIXED

    > --- Comment #3 from Wayne Davison <wayned at samba.org> ---
    > The latest git version has an option that lets you choose to include
    > nanoseconds in comparisons if you want them. Having it on by default
would
    > likely cause far too many headaches for various backup solutions that
use an
    > older filesystem (e.g. ext3) that doesn't support nanoseconds.

Thanks for the patch!

Just FYI, this comment is true but incomplete---the scenario I was
describing was straight ext4-to-ext4 copies and/or backups.  The
timestamp problem I currently see with those is because rsync was
throwing away the nanosecond information until this patch, even though
both ends supported it.

Anyone who's used dirvish, or presumably similar tools such as
rsnapshot, from and to ext4 or other ns-supporting filesystems, will
be bitten by the problem of non-ns vs ns timestamps bloating backups
and breaking hardlinks, either when they manually use --modify-window=-1,
or when this becomes the default.

I'd still love to see either some in-rsync workaround that can be left
in place*, or (second best) some clever by-hand one-time workaround
that uses rsync just once to update all those timestamps while not
breaking the hardlinks (presumably -not- using --link-dest for those),
or (third best) some not-rsync-at-all solution that basically does
a giant ls at the source and a giant touch at the destination.  Yes,
I know that none of those can fix up backed-up files that are no
longer in the same place in the source; those at least won't bloat/
unhardlink later backups.

This solution is likely going to have to be reinvented repeatedly by
people running backups, so working out the right way to do it and then
putting it somewhere other rsync users will find it is likely a useful
exercise.  Before I solve it by hand for myself, it'd be useful to
know if it's likely rsync will come up with some way to solve it for
me.

* E.g., (a) "If modify-window is -1, and --link-dest is in use, and
everything else about the file or directory matches -except- the
timestamp, then update the timestamp," and (b) "Provide a switch to
turn off this heuristic after I'm sure my backups are okay."  (The
intent of (b) is to catch later slight changes of timestamp but
actually record them as -separate files-, which is important so
older snapshots don't magically change out from under you if something
updates a timestamp by a fraction of a second without otherwise
changing anything, which does happen and can occasionally be -very-
important to know about when tracking down issues.)

> Date: Sun, 24 Jan 2016 15:43:20 -0800
    > From: Wayne Davison <wayne at thedavisons.net>

A couple questions below; please bear with me.

    > No, if you do a ext4 -> ext4 copy, rsync has set the matching ns
info for
    > transferred files since 3.1.0. There was a case prior to rsync 3.1.2
where
    > a brand-new file transferred in the same second it was created
wouldn't get
    > the right ns value because rsync was optimizing away the time-set if
the
    > file's mod-time matched in the integer part (3.1.2 fixed that).

Oh, I see what happened.  My problem is that no Ubuntu LTS before
14.04 had rsync 3.1.0 or newer, and the original capability took more
than four years to make it into a released rsync version, if I'm
reading the release notes correctly.*  Unfortunately, that means
the vast majority of my machine base predates the fix, including the
machine hosting the backups.  I can obviously install newer rsyncs,
but that gives me a big installed base of pre-fix data that I'm going
to have to fix, and no more rsync security updates unless I track them
manually.  Yet I'd rather do this now, so I'm future-proofed, than
be badly surprised some years down the road when this rsync behavior
becomes the default, and to keep the problem from continuing to get worse.

* I may have tried 3.1.0 at some point and then realized the problems
  it'd give me for backups and didn't install it everywhere, pending
  a better fix; this is starting to ring bells.  I'm really surprised
  that the initial patch of Sep 7, 2009 never made it into a released
  rsync until 3.1.0 of Sep 28 2013; that's a four-year delay, and
  explains why I obviously never tried this out until perhaps an
  experiment with 3.1.0, and no doubt I didn't want to run a private
  version that wouldn't get security updates.  Ubuntu rsync versions:
  10.04 has 3.0.7 proto 30; 12.04 has 3.0.9 p 30; 14.04 has 3.1.0 p 31.)

So what it looks like is that the capability to transfer ns times at
all existed in CVS but not released since 2009, in released since
2013, and in an Ubuntu LTS since 2014.  And the current patch -seems-
to be an optimization that avoids -comparison- if the ns times match,
but that only affects speed---it doesn't change what gets written in
any event, just how fast.  Right?  But actually I think it changes
behavior besides that---see my test case below.

    > Beginning with this patch you can run rsync -aiv --checksum - at -1 and
have
    > it fix the full mod-time on any matching files it finds. But for most
newer
    > backups, the ns time will already be set correctly (as long as it was
    > created using a new enough rsync and protocol 31). If someone has a
large
    > link-dest hierarchy that predates 3.1.0, then you could be sharing
    > hard-linked matching files from back before the ns info was included
(the
    > older files would all have 0 for the ns value).

Wow, that -c really hurts.  If one wanted to live dangerously---with
the assumption that two files that otherwise match in all metadata
(including obviously length :) but whose timestamps differ in that one
has integer seconds and the other has the same integer seconds but
also nanoseconds, can rsync readjust the dates, without doing a full
checksum?  If not, I may write such a tool, or do it the (very) slow
way and have rsync re-checksum a few terabytes of my backups... :)
[Might find some bitrot that way, of course.]

Also, I actually -can't- use that command to fix my snapshots,
because (if I understand correctly), it will -alter- my existing
snapshots to match the -current- contents of files, destroying them
---I'll no longer be able to go back in time to a previous version.
I only want to update ns times on files in the older snapshots
if and only if changing integer times to ns times would be the
only modification.  I think rsync -ac - at -1 will do far more, yes?

As for - at -1, that introduces a surprising change in behavior when I
try it.  I'm unsure if it's intended, though I think it is.  But it
will -definitely- break my hardlinks and bloat the backups if I try
it without readjusting the dates in --link-dest directories (e.g.,
previous snapshots).  I find that specifying - at -1 copies the ns
timestamp from the source to the destination even if the --link-dest
directory has an integer timestamp, and so I assume this is part of
the purpose of the patch?  Not just an optimization, but a change in
the way --link-dest might work.  Observe:

22:57:22 ~$ mkdir T
22:57:25 ~$ cd T
22:57:26 ~/T$ mkdir 1 2 3 4 5 6
22:57:30 ~/T$ lat() { ls -alF -i --full-time "$@"; }
22:57:49 ~/T$ touch 1/foo
22:57:53 ~/T$ ln 1/foo 1/bar
22:57:56 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
22:57:59 ~/T$ rsync -aviH 1/ 2/
sending incremental file list
.d..t...... ./
>f+++++++++ foo
hf+++++++++ bar => foo

sent 139 bytes  received 53 bytes  384.00 bytes/sec
total size is 0  speedup is 0.00
22:58:13 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 2/bar
1321176 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 2/foo
22:58:17 ~/T$ rsync -aviH --link-dest=../2 1/ 3/
sending incremental file list
.d..t...... ./

sent 89 bytes  received 19 bytes  216.00 bytes/sec
total size is 0  speedup is 0.00
22:59:05 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.013689572 -0500 2/bar
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.013689572 -0500 2/foo
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.013689572 -0500 3/bar
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.013689572 -0500 3/foo
22:59:10 ~/T$ touch --date="2016-01-24 22:57:53 -0500" 2/bar
23:00:07 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.000000000 -0500 2/bar
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.000000000 -0500 2/foo
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.000000000 -0500 3/bar
1321176 -rw-r--r-- 4 user user 0 2016-01-24 22:57:53.000000000 -0500 3/foo
23:00:09 ~/T$ rsync -aviH --link-dest=../2 1/ 4/
sending incremental file list
.d..t...... ./

sent 89 bytes  received 19 bytes  216.00 bytes/sec
total size is 0  speedup is 0.00
23:00:36 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 2/bar
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 2/foo
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 3/bar
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 3/foo
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 4/bar
1321176 -rw-r--r-- 6 user user 0 2016-01-24 22:57:53.000000000 -0500 4/foo
23:00:38 ~/T$ ~/rsync-HEAD-20160124-1917GMT/rsync -aviH --link-dest ../2/ 1/ 5/
sending incremental file list
.d..t...... ./

sent 89 bytes  received 19 bytes  216.00 bytes/sec
total size is 0  speedup is 0.00
23:00:56 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 2/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 2/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 3/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 3/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 4/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 4/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 5/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 5/foo
23:00:59 ~/T$ ~/rsync-HEAD-20160124-1917GMT/rsync - at -1 -aviH --link-dest
../2/ 1/ 6/
sending incremental file list
.d..t...... ./
>f..t...... bar
hf+++++++++ foo => bar

sent 136 bytes  received 50 bytes  372.00 bytes/sec
total size is 0  speedup is 0.00
23:01:13 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 2/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 2/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 3/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 3/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 4/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 4/foo
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 5/bar
1321176 -rw-r--r-- 8 user user 0 2016-01-24 22:57:53.000000000 -0500 5/foo
1321177 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 6/bar
1321177 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 6/foo
23:01:16 ~/T$ rsync --version
rsync  version 3.1.0  protocol version 31
Copyright (C) 1996-2013 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
23:07:25 ~/T$ ~/rsync-HEAD-20160124-1917GMT/rsync --version
rsync  version 3.1.3dev  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
23:07:32 ~/T$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l

23:08:21 ~/T$