Hello, anybody has a clue why this happens? The scenario: Each night I create LVM snapshot of my volume which contains some virtual disk images (VirtualBox VDIs). Then I rsync the content of one of the snapshot directories to a backup server. Each file size is between 1.3 GB and 6GB. I run the following commmand: sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/excludefile -e ssh "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" "/mnt/crypteddevice/linuxX2/mit_backup" The source directory is part of the LVM snapshot. The destination directory is a truecrypt device which is already mounted. What I would expect: Only the differences of the virtual disk files should be transfered. What happens: I discover that always the whole file is transfered. Even if the original virtual machine is idle (so only some log files should have been enhaned in the virtual disk images). I do the following: - take LVM snapshot - Rsync the desired VDI file (2GB are transferred), it takes 10min - destroy LVM snapshot - take new LVM snapshot with same parameters as before - Rsync the same file (2GB are transferred), it takes 10min Am I doing something wrong? Best regards Peter
The problem with disk image as databases is that data change a lot into the file. When you run a system on a disk image, thre is not only new data appending to the end of the file, but the entire disk image is changing. Same thing would happen with large Outlook .pst databases. Rsync nor any other partial data transfer could deal with it as those files are "black boxes" seen from the backup tool. They only make sense from inside the program that create them. The best way to solve this problem is to not do the backup of the disk image, but backup from the system inside the disk image. Then, rsync can see file changes easily. Best regards, Vitorio Le 15 ao?t 08 ? 20:41, Peter P GMX a ?crit :> Hello, anybody has a clue why this happens? > > The scenario: > Each night I create LVM snapshot of my volume which contains some > virtual disk images (VirtualBox VDIs). Then I rsync the content of > one of the snapshot directories to a backup server. Each file size > is between 1.3 GB and 6GB. > I run the following commmand: > sudo -u rsyncbackup rsync -avzP --exclude-from /home/rsyncbackup/ > excludefile -e ssh "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" "/ > mnt/crypteddevice/linuxX2/mit_backup" > The source directory is part of the LVM snapshot. > The destination directory is a truecrypt device which is already > mounted. > > What I would expect: > Only the differences of the virtual disk files should be transfered. > > What happens: > I discover that always the whole file is transfered. Even if the > original virtual machine is idle (so only some log files should have > been enhaned in the virtual disk images). > I do the following: > - take LVM snapshot > - Rsync the desired VDI file (2GB are transferred), it takes 10min > - destroy LVM snapshot > - take new LVM snapshot with same parameters as before > - Rsync the same file (2GB are transferred), it takes 10min > > Am I doing something wrong? > > Best regards > Peter > > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On Fri 15 Aug 2008, Peter P GMX wrote:> > What happens: > I discover that always the whole file is transfered. Even if the > original virtual machine is idle (so only some log files should have > been enhaned in the virtual disk images). > I do the following: > - take LVM snapshot > - Rsync the desired VDI file (2GB are transferred), it takes 10min > - destroy LVM snapshot > - take new LVM snapshot with same parameters as before > - Rsync the same file (2GB are transferred), it takes 10minIt may appear that the whole file is being transferred. What does the --stats output show? The whole of both the original and the copy need to be read through to find what blocks have changed, and that can take a lot of time. It may even be faster to use --whole-file to stop the searching of changed data and simply copy the data (like you suspect is happening now). That would save 2GB of reads. Rsync tries to optimize the network traffic; if you're doing a local transfer (no network involved) rsync will automatically do --whole-file. If your network is fast, --whole-file might be the better option as well. As you give no example of how you're invoking rsync, I'm left guessing as to your situation. Paul Slootman
On Wed 20 Aug 2008, Peter P GMX wrote:> Hallo Paul,please keep replies to list messages on the list, others may also be able to help.> here is the call and the protocol for the transfer of one file. For me > it seems that the whole file is transferred, right?Nothing in the output indicates that all the file data is being transferred; only that the file is indeed being transferred. The output of --stats will help in showing how much file data is actually transferred. Paul Slootman> sudo -u rsyncbackup rsync -avzP --exclude-from > /home/rsyncbackup/excludefile -e ssh > "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" > "/mnt/crypteddevice/linuxX2/mit_backup" > receiving file list ... > 14 files to consider > mit_backup/ > mit_backup/Debian40_XXXX.vdi > 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s > 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52 > 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3% > 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s > 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50 > 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7% > 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s > 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42 > 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979 > 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11% > 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s > 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39 > 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006 > 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15% > 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s > 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54 > 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393 > 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21% > 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22% > 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s > 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54 > 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34 > 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192 > 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29% > 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s > 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33 > 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053 > 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34% > 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s > 0:02:34 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28 > 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59 538465877 > 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21 563067426 40% > 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14 574650941 41% 8.50MB/s > 0:01:34 585401475 41% 7.67MB/s 0:01:43 604962094 43% 9.70MB/s 0:01:19 > 614156492 44% 9.44MB/s 0:01:20 616733228 44% 8.78MB/s 0:01:26 623854469 > 44% 8.19MB/s 0:01:32 634887177 45% 6.37MB/s 0:01:56 643660294 46% > 6.31MB/s 0:01:56 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s > 0:01:48 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33 > 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12 698081645 > 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37 707735864 50% > 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31 743526578 53% 10.03MB/s > 0:01:03 754724412 54% 10.71MB/s 0:00:58 761665355 54% 12.06MB/s 0:00:51 > 771483521 55% 10.55MB/s 0:00:57 778832521 55% 8.18MB/s 0:01:13 786776637 > 56% 7.51MB/s 0:01:19 794628478 56% 7.73MB/s 0:01:15 800825574 57% > 6.87MB/s 0:01:24 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s > 0:01:29 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06 > 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02 862409333 > 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00 877114368 62% > 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06 891031774 63% 6.75MB/s > 0:01:12 900651352 64% 6.72MB/s 0:01:11 912028399 65% 8.22MB/s 0:00:57 > 923130092 66% 9.63MB/s 0:00:47 930375419 66% 9.34MB/s 0:00:48 945088136 > 67% 10.23MB/s 0:00:43 953608062 68% 9.57MB/s 0:00:45 960536527 68% > 8.51MB/s 0:00:49 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s > 0:00:53 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55 > 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57 > 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57 > 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55 > 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44 > 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41 > 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52 > 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48 > 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43 > 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33 > 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29 > 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57 > 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38 > 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32 > 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29 > 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19 > 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22 > 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10 > 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05 > 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02 > 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01 > 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47 > (xfer#1, to-check=12/14)
Hello Paul, here is the status of the transfer. If I understand it right, about 3% of the data was changed (literal data against matched data). It took about 1h40min to transfer ~250MB of data. But to be honest, I do not get a real clue out of this status report. E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file transfer) I end up at 40 minutes. So almost half of the complete data transfer time of this volume is needed, although this drive is rather passive and there were almost no changes on the drive between the last sync. (The machines are a AMD X2 3800 on one originating side and a Atom 1,6GHz on the destination side, Atom pulls from AMDs side). Best regards Peter receiving file list ... 13 files to consider mit_backup/Debian40.vdi 1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13) mit_backup/Laptop_D_10GB.vdi 10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13) mit_backup/RubyServer_1_9GB.vdi 2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13) mit_backup/XXXX_web.vdi 2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13) mit_backup/Ubuntu_8_04_Development.vdi 4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13) mit_backup/Ubuntu_XXXX_3GB.vdi 3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13) mit_backup/W2000_6GB.vdi 6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13) mit_backup/XXXXXXXX.vdi 4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13) Number of files: 13 Number of files transferred: 8 Total file size: 47.710.416.177 bytes Total transferred file size: 35.369.674.240 bytes Literal data: 1.181.132.240 bytes Matched data: 34.188.542.000 bytes File list size: 530 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.984.972 Total bytes received: 249.776.322 sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec total size is 47.710.416.177 speedup is 188.01 Paul Slootman schrieb:> On Wed 20 Aug 2008, Peter P GMX wrote: > > >> Hallo Paul, >> > > please keep replies to list messages on the list, others may also be > able to help. > > >> here is the call and the protocol for the transfer of one file. For me >> it seems that the whole file is transferred, right? >> > > Nothing in the output indicates that all the file data is being > transferred; only that the file is indeed being transferred. > The output of --stats will help in showing how much file data is > actually transferred. > > > Paul Slootman > > > >> sudo -u rsyncbackup rsync -avzP --exclude-from >> /home/rsyncbackup/excludefile -e ssh >> "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" >> "/mnt/crypteddevice/linuxX2/mit_backup" >> receiving file list ... >> 14 files to consider >> mit_backup/ >> mit_backup/Debian40_XXXX.vdi >> 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% 10.02MB/s >> 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s 0:02:52 >> 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 50797900 3% >> 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 4% 5.27MB/s >> 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% 5.60MB/s 0:03:50 >> 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s 0:03:15 99630301 7% >> 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 113938846 8% 7.07MB/s >> 0:02:56 125802777 9% 8.53MB/s 0:02:25 132129910 9% 7.57MB/s 0:02:42 >> 139575721 10% 7.58MB/s 0:02:41 144926021 10% 7.23MB/s 0:02:49 149718979 >> 10% 5.61MB/s 0:03:36 154452981 11% 5.32MB/s 0:03:47 160614584 11% >> 5.00MB/s 0:04:01 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s >> 0:03:58 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39 >> 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 202404006 >> 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 216901117 15% >> 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 233657909 16% 7.19MB/s >> 0:02:37 239352121 17% 6.67MB/s 0:02:49 244896643 17% 6.44MB/s 0:02:54 >> 249979132 17% 5.75MB/s 0:03:14 255669742 18% 5.23MB/s 0:03:32 262874393 >> 18% 5.58MB/s 0:03:18 275524040 19% 7.27MB/s 0:02:30 294537718 21% >> 10.58MB/s 0:01:41 306682448 21% 12.06MB/s 0:01:28 316879483 22% >> 12.74MB/s 0:01:22 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s >> 0:02:03 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54 >> 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34 >> 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 394514192 >> 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 409033762 29% >> 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 429225330 30% 7.96MB/s >> 0:01:58 443550036 31% 10.15MB/s 0:01:31 452116589 32% 9.86MB/s 0:01:33 >> 457430449 32% 8.85MB/s 0:01:43 465762610 33% 8.68MB/s 0:01:44 471632053 >> 33% 6.66MB/s 0:02:15 477048104 34% 5.91MB/s 0:02:31 483430930 34% >> 5.93MB/s 0:02:30 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s >> 0:02:34 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28 >> 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59 538465877 >> 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21 563067426 40% >> 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14 574650941 41% 8.50MB/s >> 0:01:34 585401475 41% 7.67MB/s 0:01:43 604962094 43% 9.70MB/s 0:01:19 >> 614156492 44% 9.44MB/s 0:01:20 616733228 44% 8.78MB/s 0:01:26 623854469 >> 44% 8.19MB/s 0:01:32 634887177 45% 6.37MB/s 0:01:56 643660294 46% >> 6.31MB/s 0:01:56 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s >> 0:01:48 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33 >> 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12 698081645 >> 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37 707735864 50% >> 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31 743526578 53% 10.03MB/s >> 0:01:03 754724412 54% 10.71MB/s 0:00:58 761665355 54% 12.06MB/s 0:00:51 >> 771483521 55% 10.55MB/s 0:00:57 778832521 55% 8.18MB/s 0:01:13 786776637 >> 56% 7.51MB/s 0:01:19 794628478 56% 7.73MB/s 0:01:15 800825574 57% >> 6.87MB/s 0:01:24 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s >> 0:01:29 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06 >> 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02 862409333 >> 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00 877114368 62% >> 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06 891031774 63% 6.75MB/s >> 0:01:12 900651352 64% 6.72MB/s 0:01:11 912028399 65% 8.22MB/s 0:00:57 >> 923130092 66% 9.63MB/s 0:00:47 930375419 66% 9.34MB/s 0:00:48 945088136 >> 67% 10.23MB/s 0:00:43 953608062 68% 9.57MB/s 0:00:45 960536527 68% >> 8.51MB/s 0:00:49 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s >> 0:00:53 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55 >> 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57 >> 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57 >> 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55 >> 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44 >> 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41 >> 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52 >> 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48 >> 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43 >> 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33 >> 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29 >> 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57 >> 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38 >> 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32 >> 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29 >> 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19 >> 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22 >> 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10 >> 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05 >> 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02 >> 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01 >> 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47 >> (xfer#1, to-check=12/14) >> > >
Hello does nobody has a clue about this? Best regards Peter Peter P GMX schrieb:> Hello Paul, > > here is the status of the transfer. > If I understand it right, about 3% of the data was changed (literal > data against matched data). It took about 1h40min to transfer ~250MB > of data. > But to be honest, I do not get a real clue out of this status report. > E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with > 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file > transfer) I end up at 40 minutes. So almost half of the complete data > transfer time of this volume is needed, although this drive is rather > passive and there were almost no changes on the drive between the last > sync. (The machines are a AMD X2 3800 on one originating side and a > Atom 1,6GHz on the destination side, Atom pulls from AMDs side). > > Best regards > Peter > > receiving file list ... > 13 files to consider > mit_backup/Debian40.vdi > 1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13) > mit_backup/Laptop_D_10GB.vdi > 10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13) > mit_backup/RubyServer_1_9GB.vdi > 2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13) > mit_backup/XXXX_web.vdi > 2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13) > mit_backup/Ubuntu_8_04_Development.vdi > 4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13) > mit_backup/Ubuntu_XXXX_3GB.vdi > 3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13) > mit_backup/W2000_6GB.vdi > 6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13) > mit_backup/XXXXXXXX.vdi > 4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13) > > Number of files: 13 > Number of files transferred: 8 > Total file size: 47.710.416.177 bytes > Total transferred file size: 35.369.674.240 bytes > Literal data: 1.181.132.240 bytes > Matched data: 34.188.542.000 bytes > File list size: 530 > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 3.984.972 > Total bytes received: 249.776.322 > > sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec > total size is 47.710.416.177 speedup is 188.01 > > > > Paul Slootman schrieb: >> On Wed 20 Aug 2008, Peter P GMX wrote: >> >> >>> Hallo Paul, >>> >> >> please keep replies to list messages on the list, others may also be >> able to help. >> >> >>> here is the call and the protocol for the transfer of one file. For >>> me it seems that the whole file is transferred, right? >>> >> >> Nothing in the output indicates that all the file data is being >> transferred; only that the file is indeed being transferred. >> The output of --stats will help in showing how much file data is >> actually transferred. >> >> >> Paul Slootman >> >> >> >>> sudo -u rsyncbackup rsync -avzP --exclude-from >>> /home/rsyncbackup/excludefile -e ssh >>> "192.168.0.3:/mnt/vdi_snapshot/VDI/mit_backup" >>> "/mnt/crypteddevice/linuxX2/mit_backup" >>> receiving file list ... >>> 14 files to consider >>> mit_backup/ >>> mit_backup/Debian40_XXXX.vdi >>> 0 0% 0.00kB/s 0:00:00 10235877 0% 9.69MB/s 0:02:19 21075684 1% >>> 10.02MB/s 0:02:13 27933044 2% 8.86MB/s 0:02:30 32490259 2% 7.72MB/s >>> 0:02:52 39455369 2% 6.95MB/s 0:03:10 45837776 3% 5.89MB/s 0:03:43 >>> 50797900 3% 5.44MB/s 0:04:01 53961592 3% 5.12MB/s 0:04:16 61553523 >>> 4% 5.27MB/s 0:04:07 67043585 4% 5.05MB/s 0:04:16 75011647 5% >>> 5.60MB/s 0:03:50 84224736 6% 7.00MB/s 0:03:03 89720101 6% 6.51MB/s >>> 0:03:15 99630301 7% 7.53MB/s 0:02:47 107071219 7% 7.63MB/s 0:02:44 >>> 113938846 8% 7.07MB/s 0:02:56 125802777 9% 8.53MB/s 0:02:25 >>> 132129910 9% 7.57MB/s 0:02:42 139575721 10% 7.58MB/s 0:02:41 >>> 144926021 10% 7.23MB/s 0:02:49 149718979 10% 5.61MB/s 0:03:36 >>> 154452981 11% 5.32MB/s 0:03:47 160614584 11% 5.00MB/s 0:04:01 >>> 165958612 11% 5.00MB/s 0:04:00 170811085 12% 5.01MB/s 0:03:58 >>> 175120144 12% 4.82MB/s 0:04:07 183661136 13% 5.39MB/s 0:03:39 >>> 189571426 13% 5.52MB/s 0:03:33 195568688 14% 5.78MB/s 0:03:22 >>> 202404006 14% 6.49MB/s 0:02:59 210410776 15% 6.35MB/s 0:03:02 >>> 216901117 15% 6.48MB/s 0:02:57 225794756 16% 6.94MB/s 0:02:44 >>> 233657909 16% 7.19MB/s 0:02:37 239352121 17% 6.67MB/s 0:02:49 >>> 244896643 17% 6.44MB/s 0:02:54 249979132 17% 5.75MB/s 0:03:14 >>> 255669742 18% 5.23MB/s 0:03:32 262874393 18% 5.58MB/s 0:03:18 >>> 275524040 19% 7.27MB/s 0:02:30 294537718 21% 10.58MB/s 0:01:41 >>> 306682448 21% 12.06MB/s 0:01:28 316879483 22% 12.74MB/s 0:01:22 >>> 322410223 23% 11.06MB/s 0:01:34 331382172 23% 8.38MB/s 0:02:03 >>> 344124151 24% 8.36MB/s 0:02:02 356277012 25% 8.84MB/s 0:01:54 >>> 368731867 26% 10.30MB/s 0:01:37 376902647 27% 10.49MB/s 0:01:34 >>> 382055316 27% 8.96MB/s 0:01:50 385099184 27% 6.81MB/s 0:02:24 >>> 394514192 28% 6.14MB/s 0:02:39 399166976 28% 5.31MB/s 0:03:03 >>> 409033762 29% 6.42MB/s 0:02:29 419680637 30% 8.02MB/s 0:01:58 >>> 429225330 30% 7.96MB/s 0:01:58 443550036 31% 10.15MB/s 0:01:31 >>> 452116589 32% 9.86MB/s 0:01:33 457430449 32% 8.85MB/s 0:01:43 >>> 465762610 33% 8.68MB/s 0:01:44 471632053 33% 6.66MB/s 0:02:15 >>> 477048104 34% 5.91MB/s 0:02:31 483430930 34% 5.93MB/s 0:02:30 >>> 490941562 35% 5.74MB/s 0:02:33 496540662 35% 5.70MB/s 0:02:34 >>> 501713859 35% 5.64MB/s 0:02:34 508037197 36% 5.83MB/s 0:02:28 >>> 518535763 37% 6.53MB/s 0:02:11 526524427 37% 7.08MB/s 0:01:59 >>> 538465877 38% 8.46MB/s 0:01:38 552112154 39% 10.15MB/s 0:01:21 >>> 563067426 40% 10.23MB/s 0:01:19 573336231 41% 10.73MB/s 0:01:14 >>> 574650941 41% 8.50MB/s 0:01:34 585401475 41% 7.67MB/s 0:01:43 >>> 604962094 43% 9.70MB/s 0:01:19 614156492 44% 9.44MB/s 0:01:20 >>> 616733228 44% 8.78MB/s 0:01:26 623854469 44% 8.19MB/s 0:01:32 >>> 634887177 45% 6.37MB/s 0:01:56 643660294 46% 6.31MB/s 0:01:56 >>> 651766969 46% 8.34MB/s 0:01:27 652307889 46% 6.71MB/s 0:01:48 >>> 666193629 47% 7.38MB/s 0:01:36 675491703 48% 7.49MB/s 0:01:33 >>> 684601774 49% 7.60MB/s 0:01:31 693070083 49% 9.52MB/s 0:01:12 >>> 698081645 50% 7.45MB/s 0:01:31 704963446 50% 6.90MB/s 0:01:37 >>> 707735864 50% 5.29MB/s 0:02:07 724318532 51% 7.15MB/s 0:01:31 >>> 743526578 53% 10.03MB/s 0:01:03 754724412 54% 10.71MB/s 0:00:58 >>> 761665355 54% 12.06MB/s 0:00:51 771483521 55% 10.55MB/s 0:00:57 >>> 778832521 55% 8.18MB/s 0:01:13 786776637 56% 7.51MB/s 0:01:19 >>> 794628478 56% 7.73MB/s 0:01:15 800825574 57% 6.87MB/s 0:01:24 >>> 807411368 57% 6.69MB/s 0:01:25 813464143 58% 6.34MB/s 0:01:29 >>> 822882968 58% 6.54MB/s 0:01:25 836248436 59% 8.16MB/s 0:01:06 >>> 842835485 60% 8.15MB/s 0:01:06 850384164 60% 8.52MB/s 0:01:02 >>> 862409333 61% 9.33MB/s 0:00:55 872067584 62% 8.51MB/s 0:01:00 >>> 877114368 62% 8.14MB/s 0:01:02 882553071 63% 7.58MB/s 0:01:06 >>> 891031774 63% 6.75MB/s 0:01:12 900651352 64% 6.72MB/s 0:01:11 >>> 912028399 65% 8.22MB/s 0:00:57 923130092 66% 9.63MB/s 0:00:47 >>> 930375419 66% 9.34MB/s 0:00:48 945088136 67% 10.23MB/s 0:00:43 >>> 953608062 68% 9.57MB/s 0:00:45 960536527 68% 8.51MB/s 0:00:49 >>> 971018224 69% 9.25MB/s 0:00:44 977655293 70% 7.66MB/s 0:00:53 >>> 982496601 70% 6.80MB/s 0:00:59 990341428 70% 7.09MB/s 0:00:55 >>> 996521787 71% 6.06MB/s 0:01:04 1005523756 72% 6.63MB/s 0:00:57 >>> 1011459653 72% 6.88MB/s 0:00:54 1017321634 72% 6.41MB/s 0:00:57 >>> 1024275542 73% 6.59MB/s 0:00:55 1032295357 73% 6.36MB/s 0:00:55 >>> 1035005098 74% 5.12MB/s 0:01:08 1051640925 75% 7.47MB/s 0:00:44 >>> 1058476439 75% 7.45MB/s 0:00:44 1068116134 76% 7.79MB/s 0:00:41 >>> 1072174076 76% 8.72MB/s 0:00:36 1076894984 77% 5.90MB/s 0:00:52 >>> 1086364486 77% 6.52MB/s 0:00:46 1093933058 78% 6.04MB/s 0:00:48 >>> 1100717102 78% 6.75MB/s 0:00:42 1104505412 79% 6.53MB/s 0:00:43 >>> 1109563535 79% 5.38MB/s 0:00:51 1127916628 80% 7.85MB/s 0:00:33 >>> 1134948712 81% 7.92MB/s 0:00:32 1140284922 81% 8.33MB/s 0:00:29 >>> 1148466121 82% 8.89MB/s 0:00:27 1148596687 82% 4.18MB/s 0:00:57 >>> 1160963510 83% 5.26MB/s 0:00:43 1168681452 83% 5.74MB/s 0:00:38 >>> 1175945190 84% 5.74MB/s 0:00:37 1178665232 84% 6.46MB/s 0:00:32 >>> 1189338749 85% 6.09MB/s 0:00:33 1198811725 85% 6.47MB/s 0:00:29 >>> 1207430133 86% 6.76MB/s 0:00:27 1216592101 87% 8.89MB/s 0:00:19 >>> 1223955783 87% 8.06MB/s 0:00:20 1229789052 88% 7.19MB/s 0:00:22 >>> 1255105302 89% 10.81MB/s 0:00:12 1268802266 90% 12.01MB/s 0:00:10 >>> 1279289970 91% 12.82MB/s 0:00:08 1304782634 93% 17.29MB/s 0:00:05 >>> 1325786214 94% 16.68MB/s 0:00:04 1353009354 96% 19.89MB/s 0:00:02 >>> 1361234818 97% 19.35MB/s 0:00:01 1369458218 98% 15.27MB/s 0:00:01 >>> 1379158338 98% 12.09MB/s 0:00:01 1395660800 100% 7.94MB/s 0:02:47 >>> (xfer#1, to-check=12/14) >>> >> >> >
On Tue, 2008-08-26 at 15:02 +0200, Peter P GMX wrote:> does nobody has a clue about this?As Paul said, the bottleneck appears to be disk I/O rather than network bandwidth, so you should not expect any speedup from the delta-transfer algorithm. (Even though only 3% of the data differs, rsync needs to read the entire source and destination images from disk to determine that.) --inplace and/or --whole-file might help reduce the disk I/O. Matt> Peter P GMX schrieb: > > Hello Paul, > > > > here is the status of the transfer. > > If I understand it right, about 3% of the data was changed (literal > > data against matched data). It took about 1h40min to transfer ~250MB > > of data. > > But to be honest, I do not get a real clue out of this status report. > > E.g. Laptop_D_10GB.vdi takes around 25 minutes to be transferred with > > 6,74MBytes/sec. If I divide 10G by 6,74MBytes/sec (for a complete file > > transfer) I end up at 40 minutes. So almost half of the complete data > > transfer time of this volume is needed, although this drive is rather > > passive and there were almost no changes on the drive between the last > > sync. (The machines are a AMD X2 3800 on one originating side and a > > Atom 1,6GHz on the destination side, Atom pulls from AMDs side). > > > > Best regards > > Peter > > > > receiving file list ... > > 13 files to consider > > mit_backup/Debian40.vdi > > 1395660800 100% 5.66MB/s 0:03:55 (xfer#1, to-check=11/13) > > mit_backup/Laptop_D_10GB.vdi > > 10734313984 100% 6.74MB/s 0:25:19 (xfer#2, to-check=10/13) > > mit_backup/RubyServer_1_9GB.vdi > > 2039489024 100% 8.15MB/s 0:03:58 (xfer#3, to-check=8/13) > > mit_backup/XXXX_web.vdi > > 2874171904 100% 6.17MB/s 0:07:24 (xfer#4, to-check=7/13) > > mit_backup/Ubuntu_8_04_Development.vdi > > 4367340032 100% 6.80MB/s 0:10:12 (xfer#5, to-check=6/13) > > mit_backup/Ubuntu_XXXX_3GB.vdi > > 3221238272 100% 5.61MB/s 0:09:07 (xfer#6, to-check=5/13) > > mit_backup/W2000_6GB.vdi > > 6442476032 100% 7.49MB/s 0:13:39 (xfer#7, to-check=4/13) > > mit_backup/XXXXXXXX.vdi > > 4294984192 100% 8.05MB/s 0:08:28 (xfer#8, to-check=3/13) > > > > Number of files: 13 > > Number of files transferred: 8 > > Total file size: 47.710.416.177 bytes > > Total transferred file size: 35.369.674.240 bytes > > Literal data: 1.181.132.240 bytes > > Matched data: 34.188.542.000 bytes > > File list size: 530 > > File list generation time: 0.001 seconds > > File list transfer time: 0.000 seconds > > Total bytes sent: 3.984.972 > > Total bytes received: 249.776.322 > > > > sent 3.984.972 bytes received 249.776.322 bytes 41515.14 bytes/sec > > total size is 47.710.416.177 speedup is 188.01-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part Url : http://lists.samba.org/archive/rsync/attachments/20080826/a4208b51/attachment.bin