rsync - Dec 2014 - rsync doesn't checksum for local transfers?

> You are missing the point of the checksum.  It is a verification that
> the file was assembled on the target system correctly.  The only
> post-transfer checksum that would make any sense locally would be to
> make sure that the disk stored the file correctly which would require
> a flushing of the cache and a re-reading of the file.  Rsync has no
> capability to do this whether remote or not.
yes, but indeed this could be explained more clearly in the manpage
> > """Note that rsync always verifies that each
transferred file was
> > correctly reconstructed on the receiving side by checking a
> > whole-file checksum that is generated as the file is
> > transferred"""
let me try to add some lines :

After being written to disk, for both local and remote transfers,  the 
destination file as a whole is not being re-read for checksumming. 
Checksumming is only being done for the reconstruction process: 
The checksum is calculated across the bits being received and the 
bits being read from the target file, so essentially the updated 
target file is being checksummed while it`s being written to.

is that correct ?
> 
> On 12/03/2014 09:17 PM, Shriramana Sharma wrote:
> > Hello. Please see http://unix.stackexchange.com/a/66702. I would
> > like to have confirmation whether or not rsync verifies the
> > transferred files' integrity at the target location by
checksumming
> > as advertised in the manpage:
> > 
> > """Note that rsync always verifies that each
transferred file was
> > correctly reconstructed on the receiving side by checking a
> > whole-file checksum that is generated as the file is
> > transferred"""
> > 
> > The word "always" here seems to indicate that the integrity
check
> > will happen whether for local or network transfers, but the above
> > Stack Exchange post claims otherwise. Please clarify.
> > 
> > Also, once it is assured that the check will happen *really*
> > "always", it would be useful to advertise the fact about the
> > integrity check in the website and description part of the manpage
> > itself IMO.
> > 
> > FWIW I'm using rsync 3.1.1 (latest) on openSUSE Tumbleweed.
> > 
> > Thanks.
> > 
> 
> - -- 
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			http://www.sanitarium.net/
> 	PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> 
> iEYEARECAAYFAlR/yO4ACgkQVKC1jlbQAQdevACgvdnZ0x6n0EjpAksx0rbrBSDr
> XxYAn3jCn3M04IAcZ7vbNIWKRz+5AxRe
> =wEBv
> -----END PGP SIGNATURE-----
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Correct.  If the disk fails to store the file correctly rsync will
have no idea.  This is the same as every other file copying program I
can think of.

On 12/04/2014 06:00 PM, devzero at web.de wrote:
>> You are missing the point of the checksum.  It is a verification
>> that the file was assembled on the target system correctly.  The
>> only post-transfer checksum that would make any sense locally
>> would be to make sure that the disk stored the file correctly
>> which would require a flushing of the cache and a re-reading of
>> the file.  Rsync has no capability to do this whether remote or
>> not.
> 
> yes, but indeed this could be explained more clearly in the
> manpage
> 
>>> """Note that rsync always verifies that each
transferred file
>>> was correctly reconstructed on the receiving side by checking
>>> a whole-file checksum that is generated as the file is 
>>> transferred"""
> 
> let me try to add some lines :
> 
> After being written to disk, for both local and remote transfers,
> the destination file as a whole is not being re-read for
> checksumming. Checksumming is only being done for the
> reconstruction process: The checksum is calculated across the bits
> being received and the bits being read from the target file, so
> essentially the updated target file is being checksummed while it`s
> being written to.
> 
> is that correct ?
> 
>> 
>> On 12/03/2014 09:17 PM, Shriramana Sharma wrote:
>>> Hello. Please see http://unix.stackexchange.com/a/66702. I
>>> would like to have confirmation whether or not rsync verifies
>>> the transferred files' integrity at the target location by
>>> checksumming as advertised in the manpage:
>>> 
>>> """Note that rsync always verifies that each
transferred file
>>> was correctly reconstructed on the receiving side by checking
>>> a whole-file checksum that is generated as the file is 
>>> transferred"""
>>> 
>>> The word "always" here seems to indicate that the
integrity
>>> check will happen whether for local or network transfers, but
>>> the above Stack Exchange post claims otherwise. Please
>>> clarify.
>>> 
>>> Also, once it is assured that the check will happen *really* 
>>> "always", it would be useful to advertise the fact about
the
>>> integrity check in the website and description part of the
>>> manpage itself IMO.
>>> 
>>> FWIW I'm using rsync 3.1.1 (latest) on openSUSE Tumbleweed.
>>> 
>>> Thanks.
>>> 
>> 
>> - -- 
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>> 
Kevin Korb			Phone:    (407) 252-6853
>> Systems Administrator		Internet: FutureQuest, Inc.
>> Kevin at FutureQuest.net  (work) Orlando, Florida
>> kmk at sanitarium.net (personal) Web page:
>> http://www.sanitarium.net/ PGP public key available on web site. 
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>> 
- -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2
>> 
>> iEYEARECAAYFAlR/yO4ACgkQVKC1jlbQAQdevACgvdnZ0x6n0EjpAksx0rbrBSDr 
>> XxYAn3jCn3M04IAcZ7vbNIWKRz+5AxRe =wEBv -----END PGP
>> SIGNATURE----- -- Please use reply-all for most replies to avoid
>> omitting the mailing list. To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlSA6CwACgkQVKC1jlbQAQcQ3gCdGVcgE6XDjobzgnkIom96WGrq
DlQAn2qqZD0FT4W5Rn7yobV7k4XJqn7Q
=9/N8
-----END PGP SIGNATURE-----

On Fri, Dec 5, 2014 at 4:30 AM,  <devzero at web.de>
wrote:
> After being written to disk, for both local and remote transfers,  the
> destination file as a whole is not being re-read for checksumming.
> Checksumming is only being done for the reconstruction process:
> The checksum is calculated across the bits being received and the
> bits being read from the target file, so essentially the updated
> target file is being checksummed while it`s being written to.
Some slight modifications, including clarifications from Kevin:

For remote transfers, the integrity of the data is verified by
checksumming on-the-fly at the sending/receiving rsync instances. For
local transfers, the reading and writing are done by the same rsync
instance so such integrity verification is not required unless you
have bad RAM or storage. This means that for both local and remote
transfers, the destination file as a whole is not being re-read for
checksumming.

Note that sync may be required at the destination to actually ensure
that the received files are written to storage. Also note that an
extra rsync -c with the same source/target arguments as the first time
may be done to ensure that the received/stored files are checksum-wise
identical to the source by re-reading the files. Again, this is only
useful if the kernel will actually be accessing the data from disk
such as after clearing its read-write cache.

-- 
Shriramana Sharma ???????????? ????????????

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I also want to add again that if that additional rsync --checksum run
doesn't use --itemize-changes you won't know if the file is different
for some other reason.  Here is an example of what I mean....

% rsync -vai /etc/hosts /tmp/	#initial file
sending incremental file list
> f+++++++++ hosts
sent 1,920 bytes  received 35 bytes  3,910.00 bytes/sec
total size is 1,835  speedup is 0.94
% rsync -vainc /etc/hosts /tmp/
sending incremental file list
#	no difference found
sent 54 bytes  received 12 bytes  132.00 bytes/sec
total size is 1,835  speedup is 27.80 (DRY RUN)

# edit file and change 1 character without altering size

% rsync -vainc /etc/hosts /tmp/
sending incremental file list
> fc.t...... hosts	# file has a different timestamp. different
> content
expected

sent 61 bytes  received 19 bytes  160.00 bytes/sec
total size is 1,835  speedup is 22.94 (DRY RUN)
% touch --reference=/etc/hosts /tmp/hosts	# force correct mtime
% rsync -vainc /etc/hosts /tmp/
sending incremental file list
> fc........ hosts	# something is wrong here!
sent 57 bytes  received 19 bytes  152.00 bytes/sec
total size is 1,835  speedup is 24.14 (DRY RUN)


On 12/04/2014 07:17 PM, Shriramana Sharma wrote:
> On Fri, Dec 5, 2014 at 4:30 AM,  <devzero at web.de> wrote:
>> After being written to disk, for both local and remote transfers,
>> the destination file as a whole is not being re-read for
>> checksumming. Checksumming is only being done for the
>> reconstruction process: The checksum is calculated across the
>> bits being received and the bits being read from the target file,
>> so essentially the updated target file is being checksummed while
>> it`s being written to.
> 
> Some slight modifications, including clarifications from Kevin:
> 
> For remote transfers, the integrity of the data is verified by 
> checksumming on-the-fly at the sending/receiving rsync instances.
> For local transfers, the reading and writing are done by the same
> rsync instance so such integrity verification is not required
> unless you have bad RAM or storage. This means that for both local
> and remote transfers, the destination file as a whole is not being
> re-read for checksumming.
> 
> Note that sync may be required at the destination to actually
> ensure that the received files are written to storage. Also note
> that an extra rsync -c with the same source/target arguments as the
> first time may be done to ensure that the received/stored files are
> checksum-wise identical to the source by re-reading the files.
> Again, this is only useful if the kernel will actually be accessing
> the data from disk such as after clearing its read-write cache.
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlSBBJQACgkQVKC1jlbQAQeYHACg24vsMgAXipxy0NOgN9+vhkTn
N7gAnA8nb6RT4fwbytje/ivw9yu17z7v
=zHJq
-----END PGP SIGNATURE-----