rsync - Apr 2015 - Skip based on checksum not worked as expected when using with complex filter rules.

Hi all,

I've a local repository for Debian, and I'm doing some rsync testings on
it.  See the following commands for detail:

werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -n \ 
ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\
debian-installer/binary-amd64/Packages.gz \
repo.debian.local/debian/
[snip]
dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz

sent 46 bytes  received 202 bytes  55.11 bytes/sec
total size is 20  speedup is 0.08 (DRY RUN)

werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -nc \ 
ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\
debian-installer/binary-amd64/Packages.gz \
repo.debian.local/debian/
[snip]
sent 46 bytes  received 218 bytes  58.67 bytes/sec
total size is 20  speedup is 0.08 (DRY RUN)

As you can see, if I use checksum on that package, it shown that it's 
just the same to my local copy, so there is not need to be rsynced.

But when I use a complex rsync filter rules script based on checksum and 
others to check all of the stuff based on the corresponding rules, it 
shows that the above package need to be updated, see the following for 
detail:

1- My script is named as su_dists.rsync and the content of it is as 
follows, sorry for its complex ;-( 

___________ begin su_dists.rsync ____________

rsync -amvHKLPRSB131072 -nc --delete --delete-excluded \
-f +_dists/wheezy/**Translation-en* \
-f +_dists/wheezy/**binary-all/Packages.gz \
-f +_dists/wheezy/Release* \
-f P_dists/wheezy-backports/**Translation-en* \
-f P_dists/wheezy-backports/**binary-all/Packages.gz \
-f P_dists/wheezy-backports/Release* \
-f P_dists/wheezy-proposed-updates/**Translation-en* \
-f P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \
-f P_dists/wheezy-proposed-updates/Release* \
-f +_dists/wheezy-updates/**Translation-en* \
-f +_dists/wheezy-updates/**binary-all/Packages.gz \
-f +_dists/wheezy-updates/Release* \
-f +_dists/wheezy/**binary-amd64/Packages.gz \
-f +_dists/wheezy/**installer-amd64/current/*** \
-f P_dists/wheezy-backports/**binary-amd64/Packages.gz \
-f P_dists/wheezy-backports/**installer-amd64/current/*** \
-f P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \
-f P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \
-f +_dists/wheezy-updates/**binary-amd64/Packages.gz \
-f +_dists/wheezy-updates/**installer-amd64/current/*** \
-f +_dists/wheezy/**binary-i386/Packages.gz \
-f +_dists/wheezy/**installer-i386/current/*** \
-f P_dists/wheezy-backports/**binary-i386/Packages.gz \
-f P_dists/wheezy-backports/**installer-i386/current/*** \
-f P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \
-f P_dists/wheezy-proposed-updates/**installer-i386/current/*** \
-f +_dists/wheezy-updates/**binary-i386/Packages.gz \
-f +_dists/wheezy-updates/**installer-i386/current/*** \
-f P_dists/jessie/*** \
-f P_dists/jessie-backports/*** \
-f P_dists/jessie-proposed-updates/*** \
-f P_dists/jessie-updates/*** \
-f P_pool/*** \
-f +_dists/***/ \
-f -_* ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/

___________ end su_dists.rsync ____________

2- Then, I run my above script as follows to obtain all of the files 
filtered-out according to my filter rules and checkout the ones need to 
be updated based on checksum:  

werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk -F'/' \
'/^dists/ && $NF'
dists/wheezy-updates/Release
dists/wheezy-updates/Release.gpg
dists/wheezy-updates/contrib/binary-i386/Packages.gz
dists/wheezy-updates/contrib/debian-installer/binary-amd64/Packages.gz
dists/wheezy-updates/contrib/debian-installer/binary-i386/Packages.gz
dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
dists/wheezy-updates/main/debian-installer/binary-i386/Packages.gz
dists/wheezy-updates/non-free/binary-amd64/Packages.gz
dists/wheezy-updates/non-free/binary-i386/Packages.gz
dists/wheezy-updates/non-free/debian-installer/binary-amd64/Packages.gz
dists/wheezy-updates/non-free/debian-installer/binary-i386/Packages.gz
dists/wheezy-updates/non-free/i18n/Translation-en.bz2
dists/wheezy/contrib/debian-installer/binary-all/Packages.gz
dists/wheezy/contrib/debian-installer/binary-amd64/Packages.gz
dists/wheezy/contrib/debian-installer/binary-i386/Packages.gz
dists/wheezy/non-free/debian-installer/binary-all/Packages.gz
dists/wheezy/non-free/debian-installer/binary-amd64/Packages.gz
dists/wheezy/non-free/debian-installer/binary-i386/Packages.gz

As you can see, this time, the following file is considered to be updated:

dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz

But it has been considered not need to be updated in my first run with 
the -c/--checksum option from the terminal other than the bash srcipt.

3- If I change my bash script into the following form, i.e., with the `--
no-t' option added to the args list:

___________ begin su_dists.rsync ____________

rsync -amvHKLPRSB131072 -nc --no-t --delete --delete-excluded \
-f +_dists/wheezy/**Translation-en* \
-f +_dists/wheezy/**binary-all/Packages.gz \
-f +_dists/wheezy/Release* \
-f P_dists/wheezy-backports/**Translation-en* \
-f P_dists/wheezy-backports/**binary-all/Packages.gz \
-f P_dists/wheezy-backports/Release* \
-f P_dists/wheezy-proposed-updates/**Translation-en* \
-f P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \
-f P_dists/wheezy-proposed-updates/Release* \
-f +_dists/wheezy-updates/**Translation-en* \
-f +_dists/wheezy-updates/**binary-all/Packages.gz \
-f +_dists/wheezy-updates/Release* \
-f +_dists/wheezy/**binary-amd64/Packages.gz \
-f +_dists/wheezy/**installer-amd64/current/*** \
-f P_dists/wheezy-backports/**binary-amd64/Packages.gz \
-f P_dists/wheezy-backports/**installer-amd64/current/*** \
-f P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \
-f P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \
-f +_dists/wheezy-updates/**binary-amd64/Packages.gz \
-f +_dists/wheezy-updates/**installer-amd64/current/*** \
-f +_dists/wheezy/**binary-i386/Packages.gz \
-f +_dists/wheezy/**installer-i386/current/*** \
-f P_dists/wheezy-backports/**binary-i386/Packages.gz \
-f P_dists/wheezy-backports/**installer-i386/current/*** \
-f P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \
-f P_dists/wheezy-proposed-updates/**installer-i386/current/*** \
-f +_dists/wheezy-updates/**binary-i386/Packages.gz \
-f +_dists/wheezy-updates/**installer-i386/current/*** \
-f P_dists/jessie/*** \
-f P_dists/jessie-backports/*** \
-f P_dists/jessie-proposed-updates/*** \
-f P_dists/jessie-updates/*** \
-f P_pool/*** \
-f +_dists/***/ \
-f -_* ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/

___________ end su_dists.rsync ____________

This time, I will obtain the correct file-lists that should be updated 
based on the checksum checking, see following for detail:

werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk -F'/' \
'/^dists/ && $NF'
dists/wheezy-updates/Release
dists/wheezy-updates/Release.gpg
dists/wheezy-updates/non-free/i18n/Translation-en.bz2

As you can see, for my case, there only have three files which need to be 
updated all toghether at the moment.

Any hints on the above issues?

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Any time rsync wants to do something to a file you don't think it
should be see what --itemize-changes says about it.  In fact, -v is
almost entirely useless without --itemize-changes.  In fact, if I was
in charge I would make -v include --itemize-changes.

On 04/17/2015 09:21 PM, Hongyi Zhao wrote:
> Hi all,
> 
> I've a local repository for Debian, and I'm doing some rsync
> testings on it.  See the following commands for detail:
> 
> werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -n \ 
> ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\ 
> debian-installer/binary-amd64/Packages.gz \ 
> repo.debian.local/debian/ [snip] 
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
>  sent 46 bytes  received 202 bytes  55.11 bytes/sec total size is
> 20  speedup is 0.08 (DRY RUN)
> 
> werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -nc \ 
> ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\ 
> debian-installer/binary-amd64/Packages.gz \ 
> repo.debian.local/debian/ [snip] sent 46 bytes  received 218 bytes
> 58.67 bytes/sec total size is 20  speedup is 0.08 (DRY RUN)
> 
> As you can see, if I use checksum on that package, it shown that
> it's just the same to my local copy, so there is not need to be
> rsynced.
> 
> But when I use a complex rsync filter rules script based on
> checksum and others to check all of the stuff based on the
> corresponding rules, it shows that the above package need to be
> updated, see the following for detail:
> 
> 1- My script is named as su_dists.rsync and the content of it is as
>  follows, sorry for its complex ;-(
> 
> ___________ begin su_dists.rsync ____________
> 
> rsync -amvHKLPRSB131072 -nc --delete --delete-excluded \ -f
> +_dists/wheezy/**Translation-en* \ -f
> +_dists/wheezy/**binary-all/Packages.gz \ -f
> +_dists/wheezy/Release* \ -f
> P_dists/wheezy-backports/**Translation-en* \ -f
> P_dists/wheezy-backports/**binary-all/Packages.gz \ -f
> P_dists/wheezy-backports/Release* \ -f
> P_dists/wheezy-proposed-updates/**Translation-en* \ -f
> P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/Release* \ -f
> +_dists/wheezy-updates/**Translation-en* \ -f
> +_dists/wheezy-updates/**binary-all/Packages.gz \ -f
> +_dists/wheezy-updates/Release* \ -f
> +_dists/wheezy/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy/**installer-amd64/current/*** \ -f
> P_dists/wheezy-backports/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-amd64/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy-updates/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy/**binary-i386/Packages.gz \ -f
> +_dists/wheezy/**installer-i386/current/*** \ -f
> P_dists/wheezy-backports/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-i386/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-i386/current/*** \ -f
> +_dists/wheezy-updates/**binary-i386/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-i386/current/*** \ -f
> P_dists/jessie/*** \ -f P_dists/jessie-backports/*** \ -f
> P_dists/jessie-proposed-updates/*** \ -f P_dists/jessie-updates/***
> \ -f P_pool/*** \ -f +_dists/***/ \ -f -_*
> ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/
> 
> ___________ end su_dists.rsync ____________
> 
> 2- Then, I run my above script as follows to obtain all of the
> files filtered-out according to my filter rules and checkout the
> ones need to be updated based on checksum:
> 
> werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk
-F'/'
> \ '/^dists/ && $NF' dists/wheezy-updates/Release 
> dists/wheezy-updates/Release.gpg 
> dists/wheezy-updates/contrib/binary-i386/Packages.gz 
> dists/wheezy-updates/contrib/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/contrib/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/main/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/non-free/binary-amd64/Packages.gz 
> dists/wheezy-updates/non-free/binary-i386/Packages.gz 
> dists/wheezy-updates/non-free/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/non-free/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/non-free/i18n/Translation-en.bz2 
> dists/wheezy/contrib/debian-installer/binary-all/Packages.gz 
> dists/wheezy/contrib/debian-installer/binary-amd64/Packages.gz 
> dists/wheezy/contrib/debian-installer/binary-i386/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-all/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-amd64/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-i386/Packages.gz
> 
> As you can see, this time, the following file is considered to be
> updated:
> 
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
>  But it has been considered not need to be updated in my first run
> with the -c/--checksum option from the terminal other than the bash
> srcipt.
> 
> 3- If I change my bash script into the following form, i.e., with
> the `-- no-t' option added to the args list:
> 
> ___________ begin su_dists.rsync ____________
> 
> rsync -amvHKLPRSB131072 -nc --no-t --delete --delete-excluded \ -f
> +_dists/wheezy/**Translation-en* \ -f
> +_dists/wheezy/**binary-all/Packages.gz \ -f
> +_dists/wheezy/Release* \ -f
> P_dists/wheezy-backports/**Translation-en* \ -f
> P_dists/wheezy-backports/**binary-all/Packages.gz \ -f
> P_dists/wheezy-backports/Release* \ -f
> P_dists/wheezy-proposed-updates/**Translation-en* \ -f
> P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/Release* \ -f
> +_dists/wheezy-updates/**Translation-en* \ -f
> +_dists/wheezy-updates/**binary-all/Packages.gz \ -f
> +_dists/wheezy-updates/Release* \ -f
> +_dists/wheezy/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy/**installer-amd64/current/*** \ -f
> P_dists/wheezy-backports/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-amd64/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy-updates/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy/**binary-i386/Packages.gz \ -f
> +_dists/wheezy/**installer-i386/current/*** \ -f
> P_dists/wheezy-backports/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-i386/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-i386/current/*** \ -f
> +_dists/wheezy-updates/**binary-i386/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-i386/current/*** \ -f
> P_dists/jessie/*** \ -f P_dists/jessie-backports/*** \ -f
> P_dists/jessie-proposed-updates/*** \ -f P_dists/jessie-updates/***
> \ -f P_pool/*** \ -f +_dists/***/ \ -f -_*
> ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/
> 
> ___________ end su_dists.rsync ____________
> 
> This time, I will obtain the correct file-lists that should be
> updated based on the checksum checking, see following for detail:
> 
> werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk
-F'/'
> \ '/^dists/ && $NF' dists/wheezy-updates/Release 
> dists/wheezy-updates/Release.gpg 
> dists/wheezy-updates/non-free/i18n/Translation-en.bz2
> 
> As you can see, for my case, there only have three files which need
> to be updated all toghether at the moment.
> 
> Any hints on the above issues?
> 
> Regards
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUxsxwACgkQVKC1jlbQAQcTNQCgnjBUW0D3nS0e7y/HL5lQozyw
eGIAnjokKS3M+U0IM2xKUy9xSb/n5rUa
=ZLxY
-----END PGP SIGNATURE-----

On Fri, 17 Apr 2015 21:27:56 -0400, Kevin Korb wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Any time rsync wants to do something to a file you don't think it
should
> be see what --itemize-changes says about it.  In fact, -v is almost
> entirely useless without --itemize-changes.  In fact, if I was in charge
> I would make -v include --itemize-changes.
Thanks a lot for your hints.

Based on your hints, I use the `-cniv' options combinations and do the 
extract as follows on the output of rsync:

awk '/^>fc/'

And it does the trick.

Regards
-- 
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.