I am seeing some rather strange behavior with synch of 2 directories on the same system using 2.6.2. The older file is the image of a full backup and is 29Gig in size. The new image is a slice of an incremental backup and is 101Meg in size. the command line is: time /home/wally/rsync/rsync-2.6.2 -av --rsh=rsh --backup --stats --block-size=<xxx> --write-batch=kbup1aaa /test/Kibbutz/Kbup_1.aaa /test/Kibbutz/work What I am observing in /test/Kibbutz/work is a file .Kbup_1.aaa.AZVyuT that is 35 Meg in size after an overnight run that has been going on for 14 hours. When I kill the job, I get real 817m10.062s and user 814m45.940s sys 7m23.870s. I have tried this without the --block-size statement and it goes pretty fast but the literal data is 104M with no matches. I have tried it for a variety of --block-size=<xxx> and it always stalls with very high user times. If I make the destination fedor://test/Kibbutz with a copy of the 29G file in the destination directory, it takes about 30m of real time and 9m of user time. It seems to be specific to source and destination being on the same system. Would either Wayne or Tim give me some insight into what I am doing to screw up rsync so badly?? I did similar experiments with 2.5.7 in January and didnt see behavior like this, but at that time my full backup images were only 100 Meg or so and my incremental backups were about 10 Meg. I was experimenting with building the deltas locally and distributing them with a download server for expansion of the remote targets. wally
On Fri, Jun 11, 2004 at 02:53:53PM -0400, Wallace Matthews wrote:> I am seeing some rather strange behavior with synch of 2 directories on the same system using 2.6.2. > > The older file is the image of a full backup and is 29Gig in size. The new image is a slice of an incremental > backup and is 101Meg in size. > > the command line is: > time /home/wally/rsync/rsync-2.6.2 -av --rsh=rsh --backup --stats --block-size=<xxx> --write-batch=kbup1aaa /test/Kibbutz/Kbup_1.aaa /test/Kibbutz/work > > What I am observing in /test/Kibbutz/work is a file .Kbup_1.aaa.AZVyuT that is 35 Meg in size after an overnight run that has been going on for 14 hours. When I kill the job, I get real 817m10.062s and user 814m45.940s sys 7m23.870s. > > I have tried this without the --block-size statement and it goes pretty fast but the literal data is 104M with no matches. > > I have tried it for a variety of --block-size=<xxx> and it always stalls with very high user times. > > If I make the destination fedor://test/Kibbutz with a copy of the 29G file in the destination directory, it takes about 30m of real time and 9m of user time. > > It seems to be specific to source and destination being on the same system. > > Would either Wayne or Tim give me some insight into what I am doing to screw up rsync so badly??Do you observe the same behavior without "write-batch"? -chris> > I did similar experiments with 2.5.7 in January and didnt see behavior like this, but at that time my full backup images were only 100 Meg or so and my incremental backups were about 10 Meg. > > I was experimenting with building the deltas locally and distributing them with a download server for expansion of the remote targets. > > wally > > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Thanks for the suggestion. It works fine if I remove the --write-batch from the command line. That should narrow it down for the bug fixer(s). I know that --write-batch works ok when the reference file is on a remote system. It means that until there is a fix, I have no use for the local only case. My only purpose for using it was to create delta files that I could then send to remote system(s) to create incrementals. wally -----Original Message----- From: Chris Shoemaker [mailto:c.shoemaker@cox.net] Sent: Friday, June 11, 2004 10:31 AM To: Wallace Matthews Cc: rsync@lists.samba.org Subject: Re: what am I doing wrong On Fri, Jun 11, 2004 at 02:53:53PM -0400, Wallace Matthews wrote:> I am seeing some rather strange behavior with synch of 2 directories on the same system using 2.6.2. > > The older file is the image of a full backup and is 29Gig in size. The new image is a slice of an incremental > backup and is 101Meg in size. > > the command line is: > time /home/wally/rsync/rsync-2.6.2 -av --rsh=rsh --backup --stats --block-size=<xxx> --write-batch=kbup1aaa /test/Kibbutz/Kbup_1.aaa /test/Kibbutz/work > > What I am observing in /test/Kibbutz/work is a file .Kbup_1.aaa.AZVyuT that is 35 Meg in size after an overnight run that has been going on for 14 hours. When I kill the job, I get real 817m10.062s and user 814m45.940s sys 7m23.870s. > > I have tried this without the --block-size statement and it goes pretty fast but the literal data is 104M with no matches. > > I have tried it for a variety of --block-size=<xxx> and it always stalls with very high user times. > > If I make the destination fedor://test/Kibbutz with a copy of the 29G file in the destination directory, it takes about 30m of real time and 9m of user time. > > It seems to be specific to source and destination being on the same system. > > Would either Wayne or Tim give me some insight into what I am doing to screw up rsync so badly??Do you observe the same behavior without "write-batch"? -chris> > I did similar experiments with 2.5.7 in January and didnt see behavior like this, but at that time my full backup images were only 100 Meg or so and my incremental backups were about 10 Meg. > > I was experimenting with building the deltas locally and distributing them with a download server for expansion of the remote targets. > > wally > > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On Fri, Jun 11, 2004 at 03:51:22PM -0400, Wallace Matthews wrote:> Thanks for the suggestion. It works fine if I remove the --write-batch from the command line. > That should narrow it down for the bug fixer(s). I know that --write-batch works ok when the reference file is on a remote system. > > It means that until there is a fix, I have no use for the local only case. My only purpose for using it was to create delta files that I could then send to remote system(s) to create incrementals. >Wally, You may want to try out the CVS version of rsync with my recent batch-mode rewrite patch. It has a little different (better) interface, and it does work in the case you describe. It would be interesting to see the results of your block-size measurements. -chris> wally > > -----Original Message----- > From: Chris Shoemaker [mailto:c.shoemaker@cox.net] > Sent: Friday, June 11, 2004 10:31 AM > To: Wallace Matthews > Cc: rsync@lists.samba.org > Subject: Re: what am I doing wrong > > > On Fri, Jun 11, 2004 at 02:53:53PM -0400, Wallace Matthews wrote: > > I am seeing some rather strange behavior with synch of 2 directories on the same system using 2.6.2. > > > > The older file is the image of a full backup and is 29Gig in size. The new image is a slice of an incremental > > backup and is 101Meg in size. > > > > the command line is: > > time /home/wally/rsync/rsync-2.6.2 -av --rsh=rsh --backup --stats --block-size=<xxx> --write-batch=kbup1aaa /test/Kibbutz/Kbup_1.aaa /test/Kibbutz/work > > > > What I am observing in /test/Kibbutz/work is a file .Kbup_1.aaa.AZVyuT that is 35 Meg in size after an overnight run that has been going on for 14 hours. When I kill the job, I get real 817m10.062s and user 814m45.940s sys 7m23.870s. > > > > I have tried this without the --block-size statement and it goes pretty fast but the literal data is 104M with no matches. > > > > I have tried it for a variety of --block-size=<xxx> and it always stalls with very high user times. > > > > If I make the destination fedor://test/Kibbutz with a copy of the 29G file in the destination directory, it takes about 30m of real time and 9m of user time. > > > > It seems to be specific to source and destination being on the same system. > > > > Would either Wayne or Tim give me some insight into what I am doing to screw up rsync so badly?? > > Do you observe the same behavior without "write-batch"? > -chris > > > > > I did similar experiments with 2.5.7 in January and didnt see behavior like this, but at that time my full backup images were only 100 Meg or so and my incremental backups were about 10 Meg. > > > > I was experimenting with building the deltas locally and distributing them with a download server for expansion of the remote targets. > > > > wally > > > > -- > > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html