Hello,
My question is regarding the interpretation of --stats output and
whether I have a performance issue or not.
Can the client/server architecture (rsync client on one machine, rsync
server on another machine) be far slowlier than a local rsync session ?
This is the ouput of an rsync session:
date
rsync -v --recursive --links --perms --times --group --owner --xattrs
--acls --delete --stats --human-readable \
--link-dest=/storage/daily/home/d1 \
serveur3::home /storage/daily/home/d0
date
sam. juin 9 07:10:19 CEST 2012
receiving incremental file list
./
aquota.group
aquota.user
accueil/COURRIER/RETOUR DOC. ORIGINAUX .doc
accueil/COURRIER/demande liste papiers rue du march?.doc
accueil/COURRIER/demande par courrier MME.doc
accueil/COURRIER/plusieurs dem. par t?l..doc
accueil/Document Logement/R?pertoire GT.xls
accueil/Logements Vacants/Logements vacants 2010/LOGEMENTS VACANTS 2010
JUIL ? DEC.xls
accueil/Logements Vacants/Logements vacants 2010/Logements vacants CA
2010.xls
accueil/ilot gambetta/ACCORLO SAINT AUBIN LA POSTE SANS GT.doc
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint
Cyr - Les Cyclamens.xls
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint
Cyr - Les G?raniums.xls
commercial/contentieux/PORTE VUE LOGEMENTS/PARC SAINT CYR/013 Parc Saint
Cyr - Oasis.xls
kavupdater/
kavupdater/retranslator/retranslator.log
profiles/clambany/Application
Data/Microsoft/CryptnetUrlCache/Content/7B2238AACCEDC3F1FFE8E7EB5F575EC9
profiles/clambany/Application
Data/Microsoft/CryptnetUrlCache/MetaData/7B2238AACCEDC3F1FFE8E7EB5F575EC9
sahlm/Formation EXCEL.xls
technique/5D666000
technique/EDL 2011.xls
technique/SINISTRES 2010 2011/T DOSSIERS SINISTRES DE 02 ? 2011/TABL
SINI 2010 goupil.xls
technique/a.Cl?ment/SUIVI DES DEVIS 2010.xls
technique/a.Cl?ment/SUIVI DES DEVIS 2011.xls
technique/a.Cl?ment/SUIVI DES DEVIS 2012.xls
technique/a.Teddy/EDL/EDL 2008.xls
Number of files: 1117407
Number of files transferred: 5
Total file size: 202.86G bytes
Total transferred file size: 20.67K bytes
Literal data: 1.67K bytes
Matched data: 19.00K bytes
File list size: 30.32M
File list generation time: 0.099 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 114.17K
Total bytes received: 30.69M
sent 114.17K bytes received 30.69M bytes 43.23K bytes/sec
total size is 202.86G speedup is 6586.34
sam. juin 9 07:22:11 CEST 2012
THIS IS ROUGHLY 1600 FILES / SECOND PROCESSED
*********************************************
The server is a fast machine with SCSI disks, connected through a 1Gbs
link to the client.
The client is a new dedicated backup machine with standard dedicated
SATA disks
# hdparm -t /dev/sdc
/dev/sdc:
Timing buffered disk reads: 446 MB in 3.01 seconds = 148.31 MB/sec
CPU stats during a rsync session:
on the client (top), rsync two processes are usually on top
load average 0.11 0.31 0.36
rsync process consumes between 1-50% CPU
I did some other test without --dest-link option
date
rsync --recursive --links --perms --times --group --owner --xattrs
--acls --delete --stats --human-readable \
serveur3::home /storage/monthly/home
date
Run 1 (destination directory was empty):
*****************
sam. juin 9 07:34:08 CEST 2012
Number of files: 1117487
Number of files transferred: 1003718
Total file size: 207.09G bytes
Total transferred file size: 207.09G bytes
Literal data: 207.09G bytes
Matched data: 0 bytes
File list size: 30.32M
File list generation time: 0.695 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 19.76M
Total bytes received: 207.19G
sent 19.76M bytes received 207.19G bytes 29.34M bytes/sec
total size is 207.09G speedup is 1.00
sam. juin 9 09:31:50 CEST 2012
Load average was 0.17 0.19 0.26
Run 2:
*****************
sam. juin 9 10:14:25 CEST 2012
Number of files: 1117487
Number of files transferred: 21
Total file size: 207.09G bytes
Total transferred file size: 4.14G bytes
Literal data: 124.38M bytes
Matched data: 4.01G bytes
File list size: 30.32M
File list generation time: 0.054 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.51M
Total bytes received: 155.86M
sent 1.51M bytes received 155.86M bytes 390.99K bytes/sec
total size is 207.09G speedup is 1315.90
sam. juin 9 10:21:07 CEST 2012
THIS IS ROUGHLY 2700 FILES / SECOND PROCESSED
*********************************************
I did a third test on my home machine which is using rsync for backup on
a *locally attached USB disk* (sdc):
rsync options: '-a --delete --delete-excluded --one-file-system --stats
--human-readable'
# hdparm -t /dev/sdc1
/dev/sdc1:
Timing buffered disk reads: 70 MB in 3.02 seconds = 23.17 MB/sec
dim. juin 10 08:52:55 CEST 2012
Number of files: 107883
Number of files transferred: 0
Total file size: 564.75G bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 2.40M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.40M
Total bytes received: 2.27K
sent 2.40M bytes received 2.27K bytes 177.86K bytes/sec
total size is 564.75G speedup is 235206.08
dim. juin 10 08:53:08 CEST 2012
THIS IS ROUGHLY 107000 FILES / SECOND PROCESSED
***********************************************
40 TIMES FASTER !!!!
Thanks a lot for your kind advise
