thr3ads.net - rsync - file contents cause rsync to fail (with certains args and dir structure) [Mar 2018]

If this information is useful, please help other people find it:
Share via:

xftroxgpx

2018-Mar-04 13:57 UTC

file contents cause rsync to fail (with certains args and dir structure)

script to reproduce:

#!/bin/bash

#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz
#tested to work ok      : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz

if test "$1" == "clean"; then
  rm -vrf destdir sourcedir sourcedir2 sourcedir3
  exit 0
fi

echo '!! test 1:'

mkdir -p destdir
mkdir -p sourcedir/a
#one \0 followed by a non-\0 (so, using a space) required:
echo -ne '\0 ' > sourcedir/a/b
#non zero size file required:
echo -ne 'c' > sourcedir/c
echo 'sourcedir' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b":
No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

# ^ this happens first and any subsequent times!

echo '!! test 2:'

mkdir -p sourcedir2/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir2/c

#in order to see this error: (for file 'c')

#non-zero file size required and it must be prefixed by '.' aka dot
#echo -ne '1' > sourcedir2/.d
#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go

#otherwise, the error is for file "a/b"

echo 'sourcedir2' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b":
No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1]

echo '!! test 3:' #same as 2 but an extra file '.d' exists!

mkdir -p sourcedir3/a
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/a/b
#one \0 followed by a non-\0 (so, using an M) required:
echo -ne '\0M' > sourcedir3/c

#non-zero file size required and it must be prefixed by '.' aka dot
echo -ne '1' > sourcedir3/.d
echo 'sourcedir3' >/tmp/filesfrom.lst.tmp
rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/
#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c":
No such file or directory (2)
#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

A pristine copy of this script can be found here:
https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go

Just run ./go to start all 3 tests.

All 3 tests should fail with:
ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz
ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz

All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz
(aka rsync  version 3.1.2  protocol version 31)


Cheers!

Sent with ProtonMail Secure Email.

Dave Gordon

2018-Mar-04 22:17 UTC

head link

file contents cause rsync to fail (with certains args and dir structure)

Quite strange at first sight that the failure should depend on the files
containing NULs!

But I've reproduced it on both Ubuntu and OpenSUSE with d73762e
"Preparing for release of 3.1.3".

The problem remains even if you drop the --checksum or --delay-updates options
from the command line, but goes away if you don't use both
"--sparse" and "--preallocate" together.

So it looks like a bad interaction of these options, probably with the sparse
handling being affected by the NULs in the files.



HTH,

.Dave.



Sent using Zoho Mail






---- On Sun, 04 Mar 2018 14:57:24 +0100 xftroxgpx via rsync &lt;rsync at
lists.samba.org&gt; wrote ----




script to reproduce: 

 

#!/bin/bash 

 

#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 

#tested to work ok : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 

 

if test "$1" == "clean"; then 

 rm -vrf destdir sourcedir sourcedir2 sourcedir3 

 exit 0 

fi 

 

echo '!! test 1:' 

 

mkdir -p destdir 

mkdir -p sourcedir/a 

#one \0 followed by a non-\0 (so, using a space) required: 

echo -ne '\0 ' &gt; sourcedir/a/b 

#non zero size file required: 

echo -ne 'c' &gt; sourcedir/c 

echo 'sourcedir' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 

 

# ^ this happens first and any subsequent times! 

 

echo '!! test 2:' 

 

mkdir -p sourcedir2/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir2/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir2/c 

 

#in order to see this error: (for file 'c') 

 

#non-zero file size required and it must be prefixed by '.' aka dot 

#echo -ne '1' &gt; sourcedir2/.d 

#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go 

 

#otherwise, the error is for file "a/b" 

 

echo 'sourcedir2' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1]

 

echo '!! test 3:' #same as 2 but an extra file '.d' exists! 

 

mkdir -p sourcedir3/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir3/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir3/c 

 

#non-zero file size required and it must be prefixed by '.' aka dot 

echo -ne '1' &gt; sourcedir3/.d 

echo 'sourcedir3' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 

 

A pristine copy of this script can be found here:
https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go

 

Just run ./go to start all 3 tests. 

 

All 3 tests should fail with: 

ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 

 

All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 

(aka rsync version 3.1.2 protocol version 31) 

 

 

Cheers! 

 

Sent with ProtonMail Secure Email. 

 

 

 

-- 

Please use reply-all for most replies to avoid omitting the mailing list. 

To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html 






-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20180304/5f2c2f1b/attachment.html>

Dave Gordon

2018-Mar-05 10:14 UTC

head link

file contents cause rsync to fail (with certains args and dir structure)

Problem was introduced with this commit:



commit f3873b3d88b61167b106e7b9227a20147f8f6197

Author: Wayne Davison &lt;wayned at samba.org&gt;

Date:   Mon Oct 10 11:49:50 2016 -0700



    Support --sparse combined with --preallocate or --inplace.

   

    The new code tries to punch holes in the destination file using newer

    Linux fallocate features. It also supports a --whole-file + --sparse +

    --inplace copy on any filesystem by truncating the destination file.



.Dave.



Sent using Zoho Mail






---- On Sun, 04 Mar 2018 23:17:23 +0100 Dave Gordon via rsync &lt;rsync at
lists.samba.org&gt; wrote ----




Quite strange at first sight that the failure should depend on the files
containing NULs!

But I've reproduced it on both Ubuntu and OpenSUSE with d73762e
"Preparing for release of 3.1.3".

The problem remains even if you drop the --checksum or --delay-updates options
from the command line, but goes away if you don't use both
"--sparse" and "--preallocate" together.

So it looks like a bad interaction of these options, probably with the sparse
handling being affected by the NULs in the files.



HTH,

.Dave.



Sent using Zoho Mail






---- On Sun, 04 Mar 2018 14:57:24 +0100 xftroxgpx via rsync &lt;rsync at
lists.samba.org&gt; wrote ----












-- 

Please use reply-all for most replies to avoid omitting the mailing list. 

To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


script to reproduce: 



#!/bin/bash 



#tested to fail as below: ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

#tested to fail as below: ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 

#tested to work ok : ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 



if test "$1" == "clean"; then 

rm -vrf destdir sourcedir sourcedir2 sourcedir3 

exit 0 

fi 



echo '!! test 1:' 



mkdir -p destdir 

mkdir -p sourcedir/a 

#one \0 followed by a non-\0 (so, using a space) required: 

echo -ne '\0 ' &gt; sourcedir/a/b 

#non zero size file required: 

echo -ne 'c' &gt; sourcedir/c 

echo 'sourcedir' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir/a/b":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 



# ^ this happens first and any subsequent times! 



echo '!! test 2:' 



mkdir -p sourcedir2/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir2/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir2/c 



#in order to see this error: (for file 'c') 



#non-zero file size required and it must be prefixed by '.' aka dot 

#echo -ne '1' &gt; sourcedir2/.d 

#XXX: ^ (un)comment, don't forget to ./go clean afterwards, then ./go 



#otherwise, the error is for file "a/b" 



echo 'sourcedir2' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/a/b":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3pre1]



echo '!! test 3:' #same as 2 but an extra file '.d' exists! 



mkdir -p sourcedir3/a 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir3/a/b 

#one \0 followed by a non-\0 (so, using an M) required: 

echo -ne '\0M' &gt; sourcedir3/c 



#non-zero file size required and it must be prefixed by '.' aka dot 

echo -ne '1' &gt; sourcedir3/.d 

echo 'sourcedir3' &gt;/tmp/filesfrom.lst.tmp 

rsync --recursive --perms --checksum --delay-updates --numeric-ids --preallocate
--sparse --files-from=/tmp/filesfrom.lst.tmp -- ./ ./destdir/

#rsync: write failed on
"/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/destdir/sourcedir2/c":
No such file or directory (2)

#rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] 



A pristine copy of this script can be found here:
https://github.com/xftroxgpx/a3/blob/37ebff3e0fe9d294aeec899a082dc2c51c486eb4/system/Z575/OSes/3archlinux/on_baremetal/filesystem_now/archlinux/home/xftroxgpx/sandbox/rsync/nsfod_issues/try2/go



Just run ./go to start all 3 tests. 



All 3 tests should fail with: 

ArchLinux's rsync-3.1.3-1-x86_64.pkg.tar.xz 

ArchLinux's rsync-3.1.3pre1-1-x86_64.pkg.tar.xz 



All 3 tests will succeed with ArchLinux's rsync-3.1.2-8-x86_64.pkg.tar.xz 

(aka rsync version 3.1.2 protocol version 31) 





Cheers! 



Sent with ProtonMail Secure Email. 







-- 

Please use reply-all for most replies to avoid omitting the mailing list. 

To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync

Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html 






-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20180305/ec6a83d9/attachment.html>

Possibly Parallel Threads

Search for more seemingly similar threads

rsync - Mar 2018 - file contents cause rsync to fail (with certains args and dir structure)

file contents cause rsync to fail (with certains args and dir structure)

file contents cause rsync to fail (with certains args and dir structure)

file contents cause rsync to fail (with certains args and dir structure)

Possibly Parallel Threads