CentOS - Jan 2020 - [semi-OT] C7 Possible bug but I can't determine what tool has the problem

Hi all,

I'm writing a script that uses rsync to sync 2 dirs on C7.
I noticed a strange behaviour.

I have 2 dir: src and dest. In src dir I generate a testfile with "dd 
if=/dev/zero of=testfile bs=1M count=100" and when I run "du -h 
testfile" I get the correct result.
Then I sync src/ to dest/ using "rsync -avS? src/ dest/", all ok but 
when I run "du -h dest/testfile" I get 0 and if I run "du -b 
dest/testfile" I get the correct size in bytes.

I made several test to see what happens and noticed that removing -S 
(--sparse) from rsync command problem does not occour.
In another test, thinking about a problem on 0ed file, I tried file 
generation using /dev/urandom then /dev/zero and running rsync -avS the 
problem is disappeared.

This not seems to be a CentOS 7 related problem. I tried also on Fedora 
31 and get the same problem.


I wrote a simple bash script to replicate the? problem:

#!/bin/bash
mkdir src
mkdir dest
cd src
#dd if=/dev/urandom of=testfile bs=1M count=100
dd if=/dev/zero of=testfile bs=1M count=100
echo "src/testfile size:"
du -h testfile
du -b testfile
cd ..
rsync -avS? src/ dest/ > /dev/null
echo "dest/testfile size after rsync --sparse:"
du -h dest/testfile
du -b dest/testfile
rm -f dest/testfile
echo "dest/testfile size after rsync:"
rsync -av? src/ dest/ > /dev/null
du -h dest/testfile
du -b dest/testfile

There is a bug in rsync or in du or something else?

Thanks in advance.

On Wed, Jan 15, 2020 at 10:18 AM Alessandro Baggi <
alessandro.baggi at gmail.com> wrote:
> I made several test to see what happens and noticed that removing -S
> (--sparse) from rsync command problem does not occour.
> In another test, thinking about a problem on 0ed file, I tried file
> generation using /dev/urandom then /dev/zero and running rsync -avS the
> problem is disappeared.
>
https://wiki.archlinux.org/index.php/Sparse_file#Creating_sparse_files

In short, rsync is being told to create sparse files with the -S flag, so
it does.  Could you share what you did with the urandom then zero test you
mentioned? I'm curious what exact sequence of commands you used.

Il 15/01/20 17:51, Jon Pruente ha scritto:
> On Wed, Jan 15, 2020 at 10:18 AM Alessandro Baggi <
> alessandro.baggi at gmail.com> wrote:
>
>> I made several test to see what happens and noticed that removing -S
>> (--sparse) from rsync command problem does not occour.
>> In another test, thinking about a problem on 0ed file, I tried file
>> generation using /dev/urandom then /dev/zero and running rsync -avS the
>> problem is disappeared.
>>
> https://wiki.archlinux.org/index.php/Sparse_file#Creating_sparse_files
>
> In short, rsync is being told to create sparse files with the -S flag, so
> it does.  Could you share what you did with the urandom then zero test you
> mentioned? I'm curious what exact sequence of commands you used.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
Hi Jon,
I wrote in the first mail the script with the current order of command 
that I used. Try to run in a bash script and you will see the result.

If not my sequence is:

dd if=/dev/zero of=src/testfile bs=1M count=100
rsync -avS src/ dest/
du -h dest/testfile
du -b dest/testfile

for urandom:

dd if=/dev/urandom of=src/testfile bs=1M count=100
rsync -avS src/ dest/
du -h dest/testfile
du -b dest/testfile

without --sparse the same as first sequence without -S option.

But why du reports 0M when with -b reports correct bytes and why this 
happens only with zeroed file?




I don't know if in the original post mail script

On 1/15/20 8:18 AM, Alessandro Baggi wrote:
> Then I sync src/ to dest/ using "rsync -avS? src/ dest/", all ok
but
> when I run "du -h dest/testfile" I get 0 and if I run "du -b
> dest/testfile" I get the correct size in bytes. 

That's not a bug, that's what sparse files are.

In POSIX systems, it's possible to treat a regular file like memory, and 
one of the things you might do with such a feature is use a file to keep 
track of the last time a user logged in.? The simplest way to so that is 
to save the time value at the offset of the user's UID.? My uid is 
556600005, so if the file weren't sparse, that one entry would create an 
enormous file, but with sparse files, the system only needs to allocate 
one block to store that value.? If a process reads that file, it will 
get all zeros from the OS until it reaches the date stored at my uid offset.

Applications can't tell whether a given set of zeros in a file are 
actual zeros on disk, or if they're simply parts of the file that 
haven't been written to, so when you tell rsync to create sparse files, 
it will do its best to identify blocks that are all zeros and simply not 
write to those on the destination.? Thus, if you use /dev/zero to create 
a file on the source and then rsync it with -S, the destination file 
will use zero blocks of storage. Naturally, that can only be true with 
files whose contents are null bytes, as you get from /dev/zero.

Il 16/01/20 02:21, Gordon Messmer ha scritto:
> On 1/15/20 8:18 AM, Alessandro Baggi wrote:
>> Then I sync src/ to dest/ using "rsync -avS? src/ dest/", all
ok but
>> when I run "du -h dest/testfile" I get 0 and if I run
"du -b
>> dest/testfile" I get the correct size in bytes. 
>
>
> That's not a bug, that's what sparse files are.
>
> In POSIX systems, it's possible to treat a regular file like memory, 
> and one of the things you might do with such a feature is use a file 
> to keep track of the last time a user logged in.? The simplest way to 
> so that is to save the time value at the offset of the user's UID.? My 
> uid is 556600005, so if the file weren't sparse, that one entry would 
> create an enormous file, but with sparse files, the system only needs 
> to allocate one block to store that value.? If a process reads that 
> file, it will get all zeros from the OS until it reaches the date 
> stored at my uid offset.
>
> Applications can't tell whether a given set of zeros in a file are 
> actual zeros on disk, or if they're simply parts of the file that 
> haven't been written to, so when you tell rsync to create sparse 
> files, it will do its best to identify blocks that are all zeros and 
> simply not write to those on the destination.? Thus, if you use 
> /dev/zero to create a file on the source and then rsync it with -S, 
> the destination file will use zero blocks of storage. Naturally, that 
> can only be true with files whose contents are null bytes, as you get 
> from /dev/zero.
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
Thank you for your answer. I appreciated it.