rsync - Nov 2022 - [Bug 15254] New: rsync performs full replication with option -I since last upgrade to version 3.2.3 protocol version 31

https://bugzilla.samba.org/show_bug.cgi?id=15254

            Bug ID: 15254
           Summary: rsync performs full replication with option -I since
                    last upgrade to version 3.2.3 protocol version 31
           Product: rsync
           Version: 3.2.0
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayne at opencoder.net
          Reporter: coffeecupfrank at gmail.com
        QA Contact: rsync-qa at samba.org
  Target Milestone: ---

Since rsync was upgraded to a more recent version (3.2.3 protocol version 31)
on my Ubuntu Linux, rsync is performing a full replication every time although
it shouldn?t because nothing changed in the source or the destination folder.

The command I used was for example:

rsync -aurvxHI --delete --log-file=/mnt/mirror1/logs/ironwolf_mirror1.log
/media/ironwolf /mnt/mirror1/IronWolf >/dev/null 2>&1

I can replicate the issue in a simple test scenario /tmp/source to /tmp/dest
via

rsync -rauvxHI --delete --stats --log-file=/tmp/rsync.log /tmp/source/
/tmp/dest/

?       Performs a full replication every time the command is being launched

rsync -rauvxH --delete --stats --log-file=/tmp/rsync.log /tmp/source/
/tmp/dest/

?       Does NOT perform a full synchronization unless there are real changes

The problem appears to be the I but that should just avoid relying on timestamp
and size only?and this was working for months in my environment as it was in a
fixed crontab running several times a day. The behaviour has definitely changed
in any case.
>From the man page:
--ignore-times, -I       don't skip files that match size and time

My system:

uname -a
Linux home 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64
x86_64 x86_64 GNU/Linux

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #1 from Frank B <coffeecupfrank at gmail.com> ---
rsync --version
rsync  version 3.2.3  protocol version 31

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

Hi.

On Sat, 26 Nov 2022 21:24:45 +0000 just subscribed for rsync-qa from bugzilla
via rsync wrote:
> I can replicate the issue in a simple test scenario /tmp/source to
/tmp/dest
> via
> rsync -rauvxHI --delete --stats --log-file=/tmp/rsync.log \
>   /tmp/source/ /tmp/dest/
> Performs a full replication every time the command is being launched
If you add verbosity (ex: -vvv) you will see:

  delta-transmission disabled for local transfer or --whole-file

and thus full replication.

But you are right, even with adding --no-whole-file or using SSH produces a
full replication with -I


rsync -i -rauxHI --no-whole-file --delete --stats --log-file=/tmp/rsync.log
/tmp/source/ /tmp/dest/  -n
>f......... foo
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 4 bytes
<snip>

rsync -i -rauxHI --delete --stats --log-file=/tmp/rsync.log /tmp/source/
localhost:/tmp/dest/  -n
<f......... foo

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
<snip>

rsync -i -rauxH --delete --stats --log-file=/tmp/rsync.log /tmp/source/
localhost:/tmp/dest/  -n

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
<snip>

PS: the -a option implies -r: you can thus suppress -r

-- 
francis

Hi.

Note: re-sending to rsync at lists.samba.org

On Sat, 26 Nov 2022 21:24:45 +0000 just subscribed for rsync-qa from bugzilla
via rsync wrote:
> I can replicate the issue in a simple test scenario /tmp/source to
/tmp/dest
> via
> rsync -rauvxHI --delete --stats --log-file=/tmp/rsync.log \
>   /tmp/source/ /tmp/dest/
> Performs a full replication every time the command is being launched
If you add verbosity (ex: -vvv) you will see:

  delta-transmission disabled for local transfer or --whole-file

and thus full replication.

But you are right, even with adding --no-whole-file or using SSH produces a
full replication with -I


rsync -i -rauxHI --no-whole-file --delete --stats --log-file=/tmp/rsync.log
/tmp/source/ /tmp/dest/  -n
>f......... foo
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 4 bytes
<snip>

rsync -i -rauxHI --delete --stats --log-file=/tmp/rsync.log /tmp/source/
localhost:/tmp/dest/  -n
<f......... foo

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
<snip>

rsync -i -rauxH --delete --stats --log-file=/tmp/rsync.log /tmp/source/
localhost:/tmp/dest/  -n

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
<snip>

PS: the -a option implies -r: you can thus suppress -r

-- 
francis

https://bugzilla.samba.org/show_bug.cgi?id=15254

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #2 from Wayne Davison <wayne at opencoder.net> ---
You need to re-read the manpage for -I. You seem to have reversed its meaning.

https://download.samba.org/pub/rsync/rsync.1#opt--ignore-times

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #3 from Frank B <coffeecupfrank at gmail.com> ---
The behavior still changed as this was working for months and clearly changed
directly after upgrading to the new binary but it's okay for me. Looks like
the
I should have been l in my case due to a copy and paste issue and different
font appearances.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #4 from Wayne Davison <wayne at opencoder.net> ---
No it didn't.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #5 from Kevin Korb <rsync at sanitarium.net> ---
I had the same knee jerk reaction as Wayne to this question.  -I means re-copy
everything (or at least re-diff everything unless --whole-file).  But I never
attempted to mix it with -u so I held my tongue.  Is it possible that in the
past -I + -u meant re-copy everything but don't overwrite anything newer and
now that exception is no longer honored?  If that is the case I am not even
sure whether this would be a regression or not.  I have always considered -I to
mean re-copy everything as an alternative to the even worse option of
--checksum.  But I can see a possible use case for it interacting with -u to
redefine "everything".

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #6 from Kevin Korb <rsync at sanitarium.net> ---
I can confirm that something did change...

# mkdir /tmp/src /tmp/dest
# touch /tmp/src/a /tmp/src/b /tmp/src/c
# rsync -vai /tmp/src/ /tmp/dest/
sending incremental file list
.d..t...... ./
>f+++++++++ a
>f+++++++++ b
>f+++++++++ c
sent 211 bytes  received 76 bytes  574.00 bytes/sec
total size is 0  speedup is 0.00
# touch /tmp/dest/b
# /usr/src/rsync-3.2.0/rsync -vai -nIu /tmp/src/ /tmp/dest/
sending incremental file list

sent 77 bytes  received 12 bytes  178.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
# /usr/src/rsync-3.2.7/rsync -vai -nIu /tmp/src/ /tmp/dest/
sending incremental file list
>f......... a
>f......... c
sent 97 bytes  received 22 bytes  238.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Guess that is up to Wayne if this is a fix or a regression.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #7 from Frank B <coffeecupfrank at gmail.com> ---
Yes, it has. The crontab was unchanged for months and directly after the update
of rsync via apt, it started performing full replications. It's clearly a
result of the new binary.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

Frank B <coffeecupfrank at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |---

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #8 from Frank B <coffeecupfrank at gmail.com> ---
Quick addition:

You can say it's a "corner case", a result of wrong assumptions
regarding u/I
or a weird usecase but you can't say "nothing has changed" since
that clearly
isn't true.

The question is: is uI supported (the documentation does not state this) and
which behavior is correct, the old one before the update or the new one - or
which behavior would you expect?

I have a working solution now but that's definitely a point to be clarified
and
at least to be updated in the documentation. There could be other users too
using this combination.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|REOPENED                    |RESOLVED

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

--- Comment #9 from Wayne Davison <wayne at opencoder.net> ---
The combo of -I with -u briefly changed to be broken but it was fixed. The -u
option means that older files on the sender are ignored. -I means that files
with the same date are TRANSFERRED.  When that was not occuring, it was a bug.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=15254

Frank B <coffeecupfrank at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|WORKSFORME                  |FIXED

--- Comment #10 from Frank B <coffeecupfrank at gmail.com> ---
Ok, so you're saying the behavior before the upgrade was a bug (not
transferring old files) and this has now been fixed in the new binary so that
it's transferring the files again. This is what I was looking for - question
answered. The documentation still doesn't cover that combination, but never
mind.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.