rsync - Mar 2018 - [Bug 13320] New: file contents cause rsync to fail (with certains args and dir structure)

https://bugzilla.samba.org/show_bug.cgi?id=13320

            Bug ID: 13320
           Summary: file contents cause rsync to fail (with certains args
                    and dir structure)
           Product: rsync
           Version: 3.1.3
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: dg32768 at zoho.eu
        QA Contact: rsync-qa at samba.org
>From xftroxgpx on mailing list, see
https://lists.samba.org/archive/rsync/2018-March/031507.html

script to reproduce:

#!/bin/bash

#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz
#tested to work ok      : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz

if test "$1" == "clean"; then
  rm -vrf destdir sourcedir sourcedir2 sourcedir3
  exit 0
fi

echo '!! test 1:'

mkdir -p destdir
mkdir -p sourcedir/a
#one \0 followed by a non-\0 (so, using a space) required:
echo -ne '\0 ' > sourcedir/a/b
#non zero size file required:
echo -ne 'c' > sourcedir/c
echo 'sourcedir' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids
--preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b":
No
such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

# ^ this happens first and any subsequent times!

echo '!! test 2:'

mkdir -p sourcedir2/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/c

#in order to see this error: (for file 'c')

#non-zero file size required and it must be prefixed by '.' aka dot
#echo -ne '1' > sourcedir2/.d
#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go

#otherwise, the error is for file "a/b"

echo 'sourcedir2' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids
--preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b":
No
such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374)
[receiver=3.1.3pre1]

echo '!! test 3:' #same as 2 but an extra file '.d' exists!

mkdir -p sourcedir3/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/c

#non-zero file size required and it must be prefixed by '.' aka dot
echo -ne '1' > sourcedir3/.d
echo 'sourcedir3' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids
--preallocate --sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c":
No such
file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

A pristine copy of this script can be found here:
https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go

Just run ./go to start all 3 tests.

All 3 tests should fail with:
ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz

All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz
(aka rsync  version 3.1.2  protocol version 31)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13320

--- Comment #1 from Dave Gordon <dg32768 at zoho.eu> ---
Bug will be triggered if:

options include both --sparse and --preallocate, AND
the second (or subsequent) file to be copied begins with one or more zero
bytes.

.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13320

--- Comment #2 from Dave Gordon <dg32768 at zoho.eu> ---
Created attachment 14018
  --> https://bugzilla.samba.org/attachment.cgi?id=14018&action=edit
Simple although probably suboptimal fix

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13320

--- Comment #3 from Dave Gordon <dg32768 at zoho.eu> ---
Problem introduced by
    commit f3873b3d88b61167b106e7b9227a20147f8f6197
    Author: Wayne Davison <wayned at samba.org>
    Date:   Mon Oct 10 11:49:50 2016 -0700

    Support --sparse combined with --preallocate or --inplace.

    The new code tries to punch holes in the destination file using newer
    Linux fallocate features. It also supports a --whole-file + --sparse +
    --inplace copy on any filesystem by truncating the destination file.

Root cause is that the new variable 'sparse_past_write' is not reset
when switching files, thus leading to miscalculation of the current
seek position in do_punch_hole().

Quick fix is to reset this variable to the current seek position on
each call to write_sparse() - probably not the most efficient fix,
but it seems to work.

.Dave.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13320

--- Comment #4 from Marcus Linsner <constantoverride+bugzilla.samba at
gmail.com> ---
Thanks for this fix Dave, worksforme!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13320

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #5 from Wayne Davison <wayne at opencoder.net> ---
This was fixed in commit 888ce058.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.