Administrator
2008-Oct-29 09:58 UTC
Is rsync copying over the entire file? Sent bytes matches actual file size.
Hi,
I am testing rsync and it looks as though it is copying over the entire
file which doesn't sound right.
System: Windows 2003
Rsync.exe
cygpopt-0.dll
cygwin1.dll
I have just tested rsync with a 571Mb zip file containing our Backup
Exec installation source.
The command I am running is:
rsync -avp --delete --progress --stats /cygdrive/c/rsync_test/
"/cygdrive/Y/" > rsync_log.txt
First I copied the file BackUpExec-11D.7170_32BIT_VERSION.zip into both
source and target and ran an rsysnc (I didn't have the progress or stats
switches on the first run).
This is what my log shows:
building file list ...
0 files...
2 files to consider
./
Number of files: 2
Number of files transferred: 0
Total file size: 584742597 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 118
File list generation time: 0.002 seconds File list transfer time: 0.000
seconds Total bytes sent: 146 Total bytes received: 32
sent 146 bytes received 32 bytes 356.00 bytes/sec total size is
584742597 speedup is 3285070.77
Then I opened up the zip and added a few extra words to a css file and
then ran another rsync.
This is the output I received (with the progress and stas switches).
building file list ...
0 files...
2 files to consider
./
BackUpExec-11D.7170_32BIT_VERSION.zip
32768 0% 0.00kB/s 0:00:00
16482304 2% 15.67MB/s 0:00:35
32735232 5% 15.59MB/s 0:00:34
50429952 8% 16.02MB/s 0:00:32
66846720 11% 15.93MB/s 0:00:31
81887232 14% 15.59MB/s 0:00:31
98074624 16% 15.57MB/s 0:00:30
110133248 18% 14.14MB/s 0:00:32
118128640 20% 12.14MB/s 0:00:37
135299072 23% 12.64MB/s 0:00:34
149061632 25% 12.07MB/s 0:00:35
164102144 28% 12.85MB/s 0:00:31
181764096 31% 15.16MB/s 0:00:25
198770688 33% 15.12MB/s 0:00:24
216334336 36% 16.02MB/s 0:00:22
235372544 40% 16.98MB/s 0:00:20
244023296 41% 14.83MB/s 0:00:22
263716864 45% 15.48MB/s 0:00:20
280264704 47% 15.24MB/s 0:00:19
297828352 50% 14.79MB/s 0:00:18
316997632 54% 17.28MB/s 0:00:15
333611008 57% 16.53MB/s 0:00:14
349667328 59% 16.42MB/s 0:00:13
364249088 62% 15.82MB/s 0:00:13
381976576 65% 15.47MB/s 0:00:12
394887168 67% 14.59MB/s 0:00:12
413597696 70% 15.22MB/s 0:00:10
429490176 73% 15.54MB/s 0:00:09
446267392 76% 15.31MB/s 0:00:08
464551936 79% 16.59MB/s 0:00:07
476577792 81% 15.00MB/s 0:00:07
495681536 84% 15.77MB/s 0:00:05
511803392 87% 15.62MB/s 0:00:04
529793024 90% 15.55MB/s 0:00:03
546209792 93% 16.60MB/s 0:00:02
563937280 96% 16.27MB/s 0:00:01
581697536 99% 16.66MB/s 0:00:00
584742598 100% 15.39MB/s 0:00:36 (1, 100.0% of 2)
Number of files: 2
Number of files transferred: 1
Total file size: 584742598 bytes
Total transferred file size: 584742598 bytes
Literal data: 584742598 bytes
Matched data: 0 bytes
File list size: 118
File list generation time: 0.002 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 584814160
Total bytes received: 48
sent 584814160 bytes
received 48 bytes
15595045.55 bytes/sec
total size is 584742598
This looks to me as though it is transferring the entire file as the
sent bytes is 584814160 and I am now confused as it should have only
transferred the changes in the zip file, although it does say that it
only received 48 bytes.
Am I reading this wrong?
Thanks for your help.
Michal Soltys
2008-Oct-29 10:34 UTC
Is rsync copying over the entire file? Sent bytes matches actual file size.
Administrator wrote:> Hi, > > [cut] > > Number of files: 2 > Number of files transferred: 1 > Total file size: 584742598 bytes > Total transferred file size: 584742598 bytes > Literal data: 584742598 bytes > Matched data: 0 bytes > File list size: 118 > File list generation time: 0.002 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 584814160 > Total bytes received: 48 > sent 584814160 bytes > received 48 bytes > 15595045.55 bytes/sec > total size is 584742598 > > > > This looks to me as though it is transferring the entire file as the > sent bytes is 584814160 and I am now confused as it should have only > transferred the changes in the zip file, although it does say that it > only received 48 bytes. > > Am I reading this wrong? > > Thanks for your help. >Locally used rsync will default to -W which causes whole files to be transferred (if they differ - also check -c option if mtime and size are not reliable in your case). But it still distinguishes between client and server - the stats are for the client - which "sent" 584814160 bytes in one direction, but "received" only 48 from the server.
Administrator
2008-Oct-29 15:31 UTC
Is rsync copying over the entire file? Sent bytes matches actual file size.
From: Michal Soltys>> Locally used rsync will default to -W which causes whole files to betransferred (if they differ - also check -c option if mtime and size are not>> reliable in your case). But it still distinguishes between client andserver - the stats are for the client - which "sent" 584814160 bytes in one>> direction, but "received" only 48 from the server.Thank you for that. I added the --no-whole-file switch and that has sorted it. Now I have another couple of questions. Question 1: I am using version 2.6.6 which shows this no whole file option in the man page: -W, --whole-file copy files whole (without rsync algorithm) --no-whole-file always use incremental rsync algorithm But someone else has a newer version (3.0.4) and the man page doesn't show this option, only the -W switch: -W, --whole-file copy files whole (without delta-xfer algorithm) But the --no-whole-file switch does work. So it is an option but not documented in the man page. Question 2: The time taken to run the command is the same with or without the switch. Why is this? I would expect the command with the no whole file switch to run faster as it only has to transfer a portion of the entire file. Thanks, Hedley ---------------- Letterpart Ltd Email Notice ------------------- This email and any files transmitted with it are confidential and may also be privileged. They are intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient please notify Letterpart Ltd immediately by return email or by calling 01737 223329. You should not copy this email or use it for any purpose nor disclose its contents to any other person. Although this email and any attachments are believed to be free of any virus, or any other defect, which might affect any computer or IT system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Letterpart Ltd for any loss or damage arising in any way from receipt or use thereof. ##Email Scanned by Trend Micro## Registered in England no 2607162 Registered Office: 16 Bell Street, Reigate, Surrey, RH2 7BG ------------End of Letterpart Ltd Email Notice ----------------- 29/10/2008
Michal Soltys
2008-Oct-29 21:04 UTC
Is rsync copying over the entire file? Sent bytes matches actual file size.
Hedley J. Phillips wrote:> Question 2: > > The time taken to run the command is the same with or without the > switch. Why is this? I would expect the command with the no whole file > switch to run faster as it only has to transfer a portion of the entire > file. >Do you mean precisely the same, or maybe just felt similar ? Either way - to know which portion(s) to transfer, rsync has to read (and process) the file on both ends, and update the destination file accordingly. http://rsync.samba.org/tech_report/node2.html With --whole-file it's just read here, write there. If the bandwidth for the transfer is large, it will likely be faster.