Hi @all! Sorry about that many questions, but after searching and reading tons different web sites, I didn't find exactly what I am searching for. So, I know that with the -z Option rsync compresses the files with gzip, than the files are transfared and at the target machine uncompressed. I want to know, is there a possibility to see how big the compressed file is, which rsync generates before being transfered? If not, is there a chance to let the file compressed on the target machine? I want to make some tests, also with the compression ratio. I made a test and transfered a 1GB file. I changed the timestamp with the touch command and rsync needs with the -c Option 42sec and without 80sec! I expected it viceversa! How can it be?! (I made the test several times) I also changed the user rights with chmod. Here the result were what I expected! Rsync needed 45sec with the -c Option, and 2sec without! Again, thank you for spending time to answer! David ____________________________________________________________________ Psssst! Schon vom neuen WEB.DE MultiMessenger geh?rt? Der kann`s mit allen: http://www.produkte.web.de/messenger/?did=3123
On Tue 27 Jan 2009, David de Lama wrote:> > So, I know that with the -z Option rsync compresses the files with gzip, than the files are transfared and at the target machine uncompressed.No, the data over the wire is compressed with the -z option; not the file.> I made a test and transfered a 1GB file. I changed the timestamp with the touch command and rsync needs with the -c Option 42sec and without 80sec! > I expected it viceversa! How can it be?! (I made the test several times) > I also changed the user rights with chmod. Here the result were what I expected! Rsync needed 45sec with the -c Option, and 2sec without!Did you flush your disk cache between each try? At both ends? Paul
On Tue 27 Jan 2009, Paul Slootman wrote:> No, the data over the wire is compressed with the -z option; not the > file.Correction: more specifically, the data between the sender and the receiver processes is compressed. Unfortunately this also happens when the transfer is local and -z happens to be passed as an option; the result is that the transfer is slowed down significantly without any benefit at all. Rsync should perhaps give a warning about the useless use of -z and maybe ignore it? Paul
@Paul: Yes, I ran sync on both machines. The same results! :( @Matt: I want to test it local, but how can I do it? I don't see the amount of data rsync do compress. _______________________________________________________________________ NUR NOCH BIS 31.01.! WEB.DE FreeDSL - Telefonanschluss + DSL f?r nur 16,37 EURO/mtl.!* http://dsl.web.de/?ac=OM.AD.AD008K13805B7069a
Tue, 27 Jan 2009 15:17:15 +0100, david.delama wrote:> Yes, I ran sync on both machines. The same results! :(sync is not enough. You will need to call a script dropcaches.sh that should contain (at least?): ------------------------------------------------------------- # dropcaches.sh # deletes all file-io caches (pagecache, dentries and inodes) sync echo 3 > /proc/sys/vm/drop_caches ------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: not available Url : http://lists.samba.org/archive/rsync/attachments/20090127/47511cd7/attachment.bin
Thanks a lot guys! I tested several files with the compression option and the compress level. But the strange thing with the timestamps is still disturbing me. I deleted the cache as Sven told me with a bash file: sync echo 3 > /proc/sys/vm/drop_caches After creating a file of 1GB I copy it with the standard rsync command. Then I made "touch -t 200901261330 test.xyz" and run this command: rsync -av --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 It takes about 80sec. Then I made "touch test.xyz" and run the rsync command with forced md5: rsync -acv --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 Now it takes only about 45sec. So I am still wondering. Am I doing s.th. wrong?! Greetings, David ____________________________________________________________________ Psssst! Schon vom neuen WEB.DE MultiMessenger geh?rt? Der kann`s mit allen: http://www.produkte.web.de/messenger/?did=3123
On Wed 28 Jan 2009, David de Lama wrote:> > But the strange thing with the timestamps is still disturbing me. > I deleted the cache as Sven told me with a bash file: > > sync > echo 3 > /proc/sys/vm/drop_cachesDid you do this between EVERY step below?> After creating a file of 1GB I copy it with the standard rsync command. > Then I made "touch -t 200901261330 test.xyz" and run this command: > rsync -av --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 > It takes about 80sec. > Then I made "touch test.xyz" and run the rsync command with forced md5: > rsync -acv --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 > Now it takes only about 45sec.Create the file, flush the cache. Copy it with rsync, flush the cache. touch the file, flush the cache. etc. Paul
On Wed, Jan 28, 2009 at 12:35:05PM +0100, David de Lama wrote:> rsync -acv --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 > Now it takes only about 45sec. > So I am still wondering. Am I doing s.th. wrong?!The -c option tells rsync to read the source file and the destination file prior to the deciding if the file needs to be transferred. So the total time taken will include the time spent doing a checksum on the sender, plus the time spent doing a checksum on the receiver, which is usually pretty slow. ..wayne..
> > But the strange thing with the timestamps is still disturbing me. > I deleted the cache as Sven told me with a bash file: > > sync > echo 3 > /proc/sys/vm/drop_cachesDid you do this between EVERY step below?> After creating a file of 1GB I copy it with the standard rsync command. > Then I made "touch -t 200901261330 test.xyz" and run this command: > rsync -av --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 > It takes about 80sec. > Then I made "touch test.xyz" and run the rsync command with forced md5: > rsync -acv --bwlimit=10000 --stats --progress --delete 192.168.222.82:"/home/test" /backup/rsync0 > Now it takes only about 45sec.Create the file, flush the cache. Copy it with rsync, flush the cache. touch the file, flush the cache. etc. ---------------------------------------------------------------- YES, the same results! The copy command takes nearly twice as long as the -c command!:( I can't explain it!!! With the copy command I get following result: =============================== receiving incremental file list test/test.xyz 1048576000 100% 15.86MB/s 0:01:03 (xfer#1, to-check=0/2) Number of files: 2 Number of files transferred: 1 Total file size: 1048576000 bytes Total transferred file size: 1048576000 bytes Literal data: 0 bytes Matched data: 1048576000 bytes File list size: 45 File list generation time: 0.013 seconds File list transfer time: 0.000 seconds Total bytes sent: 226747 Total bytes received: 129657 sent 226747 bytes received 129657 bytes 3581.95 bytes/sec total size is 1048576000 speedup is 2942.10 And with the -c command: ================= receiving incremental file list Number of files: 2 Number of files transferred: 0 Total file size: 1048576000 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 61 File list generation time: 0.009 seconds File list transfer time: 0.000 seconds Total bytes sent: 15 Total bytes received: 85 sent 15 bytes received 85 bytes 1.77 bytes/sec total size is 1048576000 speedup is 10485760.00 A question: In the copy command above, I get after receiving the incremental file list a transfer rate of 15.86MB/s. Is this the speed testing the checksums? Because I set the bwlimit to 10000. Or am I wrong? _______________________________________________________________________ NUR NOCH BIS 31.01.! WEB.DE FreeDSL - Telefonanschluss + DSL f?r nur 16,37 EURO/mtl.!* http://dsl.web.de/?ac=OM.AD.AD008K13805B7069a
> > >The first command is transferring the file and the second is not, >because the file has already been transferred. That's why the first >command is taking longer. What did you expect to see? > > >-- >MattThanks for your patience, Matt. But when I change the user rights on this file with chmod and than run rsync, it takes 2sec without -c, and 35sec with -c. Therefore I am confused! Or didn't I understand it yet! David ____________________________________________________________________ Psssst! Schon vom neuen WEB.DE MultiMessenger geh?rt? Der kann`s mit allen: http://www.produkte.web.de/messenger/?did=3123