bruce at sqls.net
2014-Feb-10 22:05 UTC
Rsync performance with large exchange database files
Clean copy. I even used the -W flag to see if it made a difference but, nope. I'm testing this same test on some of my other servers too. See if there's any common-ground I can find. On another servers (MS SQL Server) with faster disks I tried a similar test just now. There's only the C drive on this server but I used my same test file from the other server and used rsync (3.1.0) to copy the folder from one folder to another folder and it kicked off and got up to about 25MB/sec. Thing is if I just use windows to copy the same file from one folder to the other it does the whole file (3.7GB) in about 5.5 seconds (timed with my phone) so that's also a pretty massive difference. Maybe this is normal and I've just not noticed it on these other servers since they have a much smaller amount of data to backup? Still seems like some thing is wrong. I wouldn't expect the speed difference to be that huge. ------ Original Message ------ From: "Cary Lewis" <cary.lewis at gmail.com> To: bruce at sqls.net Sent: 2/10/2014 3:56:35 PM Subject: Re: Rsync performance with large exchange database files>when you were doing rsync from /cygdrive/c to /cygdrive/d was the >exchange file already there? Or was it clean copy? > >On Mon, Feb 10, 2014 at 4:03 PM, <bruce at sqls.net> wrote: >> Okay, so I've done some testing.. >> >> I created a roughly 4gb file from one of the smaller exchange >>database >> files. >> >> If I copy that to remotely to my desktop, I get about 45-50MB/sec >>read speed >> off the D (exchange database) drive. If I copy that back to the C >>drive >> (just the OS) for the Windows server it writes to the C drive at >>almost >> 100MB/sec over the network. >> >> If I copy directly from the server D drive to it's C drive using >>windows >> it's around 45MB/sec >> >> Inside cygwin using just the copy command I get about 35-45MB/sec >>transfer >> speed so there's a little hit just from cygwin. >> >> Using rsync to "sync" the file from the D drive to the C drive with >>the >> --progress option. I'm getting about 2-2.5MB/sec transfer speed >> >> The server is being used... So I've run the tests a few times >>thoughout the >> last hour or so and these are about my average numbers. >> >> Why would rsync be so much slower? Is there something I can test to >>help >> figure this out? I'm using rsync on a couple dozen Windows servers >>and it's >> been working great so I'm not sure why this one is acting weird. >> >> >> >> ------ Original Message ------ >> From: "Kevin Korb" <kmk at sanitarium.net> >> To: rsync at lists.samba.org >> Sent: 2/10/2014 10:57:08 AM >> Subject: Re: Rsync performance with large exchange database files >> >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA1 >>> >>> 3.1.0 will probably help some. >>> >>> What are the specs of the FreeBSD system? I have found that ZFS on >>> FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is the >>> minimum if dedup is disabled and 16BG of RAM for when dedup is >>>enabled. >>> >>> Also, a cache disk helps a lot. >>> >>> On 02/10/2014 10:22 AM, bruce at sqls.net wrote: >>>> >>>> >>>> >>>> ------ Original Message ------ From: bruce at sqls.net >>>> <mailto:bruce at sqls.net> To: rsync at lists.samba.org >>>> <mailto:rsync at lists.samba.org> Sent: 2/10/2014 8:38:06 AM Subject: >>>> Rsync performance with large exchange database files >>>> >>>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to >>>>> backup all the servers at my day job. This works pretty good >>>>> overall but on one server it's not working so well :) >>>>> >>>>> We have an Exchange 2003 server with 4 separate mail store >>>>> databases. One of them is roughly 900GB the others are ~200GB, >>>>> ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to >>>>> find the differences in the files. On the Windows server where >>>>> rsync is kicked off there's very little CPU or RAM usage for the >>>>> rsync client. On the server rsync (rsyncd, no ssh) is using >>>>> around 70-85% of a cpu (well, half a cpu due to hyper threading). >>>>> I'm using VSS on the windows server to take a snapshot and expose >>>>> it then running rsync from that to avoid locking issues. >>>>> >>>>> Is there anything I should check to help narrow down "problems?" >>>>> or any settings I should try that could help speed things up >>>>> any? >>>>> >>>>> Below is the final output of the last two rsync runs to give you >>>>> an idea. It's taking 30-40+ hours to finish even though it's >>>>> only transferring 80-160GB of change. Right now I'm testing this >>>>> against a local rsync server so it should get pretty fast network >>>>> performance. Eventually it will be moved to our off-site backup >>>>> but that connection is still pretty fast (20 MBbit) and the >>>>> backup is only hitting 800-1000 Kbytes/sec. >>>>> >>>>> >>>>> Number of files: 19 Number of files transferred: 6 Total file >>>>> size: 1265.74G bytes Total transferred file size: 1057.06G bytes >>>>> Literal data: 160.67G bytes Matched data: 896.39G bytes File list >>>>> size: 482 File list generation time: 0.001 seconds File list >>>>> transfer time: 0.000 seconds Total bytes sent: 160.71G Total >>>>> bytes received: 73.74M >>>>> >>>>> sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total >>>>> size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0, >>>>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, >>>>> line=1052): about to call exit(0) >>>>> >>>>> real 2833m1.324s user 2225m55.906s sys 45m10.015s >>>>> >>>>> Number of files: 11 Number of files transferred: 6 Total file >>>>> size: 1268.78G bytes Total transferred file size: 1251.04G bytes >>>>> Literal data: 83.43G bytes Matched data: 1167.61G bytes File list >>>>> size: 216 File list generation time: 1.360 seconds File list >>>>> transfer time: 0.000 seconds Total bytes sent: 83.48G Total bytes >>>>> received: 87.25M >>>>> >>>>> sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec total >>>>> size is 1268.78G speedup is 15.18 >>>>> >>>>> real 1745m5.647s user 1129m14.000s sys 39m58.875s >>>>> >>>>> Thanks (in advance) for the help :) >>>> >>>> >>>> rsync options I'm using on the client are : -rltihv --progress >>>> --stats --inplace --modify-window=1 >>>> >>>> On the windows client I'm using cygwin + rsync 3.0.9 but I'm going >>>> to test 3.1.0 there and see if there's a difference. >>>> >>>> On the server it's rsync 3.1.0 running rsyncd. >>>> >>>> Perhaps useful bit of information :). >>>> >>>> >>>> >>> >>> - -- >>> >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ >>> Kevin Korb Phone: (407) 252-6853 >>> Systems Administrator Internet: >>> FutureQuest, Inc. Kevin at FutureQuest.net (work) >>> Orlando, Florida kmk at sanitarium.net (personal) >>> Web page: http://www.sanitarium.net/ >>> PGP public key available on web site. >>> >>> >>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2.0.22 (GNU/Linux) >>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >>> >>> iEYEARECAAYFAlL5BOQACgkQVKC1jlbQAQeEVACgl7LEtMrX52lMqNukkCn3+poR >>> k/oAoOdugEyjn5AeP54zvbwazhjyIKc4 >>> =ll+E >>> -----END PGP SIGNATURE----- >>> -- >>> Please use reply-all for most replies to avoid omitting the mailing >>>list. >>> To unsubscribe or change options: >>> https://lists.samba.org/mailman/listinfo/rsync >>> Before posting, read: >>>http://www.catb.org/~esr/faqs/smart-questions.html >> >> >> -- >> Please use reply-all for most replies to avoid omitting the mailing >>list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: >>http://www.catb.org/~esr/faqs/smart-questions.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rsync is known to be pretty inefficient on local copies (-W is forced there btw) and cygwin doesn't really help with that either. Essentially, when not networking rsync isn't much smarter than cp but it has a ton of extra overhead. Also, maybe you want unison since there is a native Windows version of it? On 02/10/2014 05:05 PM, bruce at sqls.net wrote:> > Clean copy. I even used the -W flag to see if it made a difference > but, nope. > > I'm testing this same test on some of my other servers too. See if > there's any common-ground I can find. > > On another servers (MS SQL Server) with faster disks I tried a > similar test just now. There's only the C drive on this server but > I used my same test file from the other server and used rsync > (3.1.0) to copy the folder from one folder to another folder and it > kicked off and got up to about 25MB/sec. Thing is if I just use > windows to copy the same file from one folder to the other it does > the whole file (3.7GB) in about 5.5 seconds (timed with my phone) > so that's also a pretty massive difference. > > Maybe this is normal and I've just not noticed it on these other > servers since they have a much smaller amount of data to backup? > Still seems like some thing is wrong. I wouldn't expect the speed > difference to be that huge. > > > > ------ Original Message ------ From: "Cary Lewis" > <cary.lewis at gmail.com> To: bruce at sqls.net Sent: 2/10/2014 3:56:35 > PM Subject: Re: Rsync performance with large exchange database > files > >> when you were doing rsync from /cygdrive/c to /cygdrive/d was >> the exchange file already there? Or was it clean copy? >> >> On Mon, Feb 10, 2014 at 4:03 PM, <bruce at sqls.net> wrote: >>> Okay, so I've done some testing.. >>> >>> I created a roughly 4gb file from one of the smaller exchange >>> database files. >>> >>> If I copy that to remotely to my desktop, I get about >>> 45-50MB/sec read speed off the D (exchange database) drive. If >>> I copy that back to the C drive (just the OS) for the Windows >>> server it writes to the C drive at almost 100MB/sec over the >>> network. >>> >>> If I copy directly from the server D drive to it's C drive >>> using windows it's around 45MB/sec >>> >>> Inside cygwin using just the copy command I get about >>> 35-45MB/sec transfer speed so there's a little hit just from >>> cygwin. >>> >>> Using rsync to "sync" the file from the D drive to the C drive >>> with the --progress option. I'm getting about 2-2.5MB/sec >>> transfer speed >>> >>> The server is being used... So I've run the tests a few times >>> thoughout the last hour or so and these are about my average >>> numbers. >>> >>> Why would rsync be so much slower? Is there something I can >>> test to help figure this out? I'm using rsync on a couple dozen >>> Windows servers and it's been working great so I'm not sure why >>> this one is acting weird. >>> >>> >>> >>> ------ Original Message ------ From: "Kevin Korb" >>> <kmk at sanitarium.net> To: rsync at lists.samba.org Sent: 2/10/2014 >>> 10:57:08 AM Subject: Re: Rsync performance with large exchange >>> database files >>> > 3.1.0 will probably help some. > > What are the specs of the FreeBSD system? I have found that ZFS on > FreeBSD is extremely RAM hungry. In my experience 8GB of RAM is > the minimum if dedup is disabled and 16BG of RAM for when dedup is >>>>> enabled. > > Also, a cache disk helps a lot. > > On 02/10/2014 10:22 AM, bruce at sqls.net wrote: >>>>>> >>>>>> >>>>>> >>>>>> ------ Original Message ------ From: bruce at sqls.net >>>>>> <mailto:bruce at sqls.net> To: rsync at lists.samba.org >>>>>> <mailto:rsync at lists.samba.org> Sent: 2/10/2014 8:38:06 AM >>>>>> Subject: Rsync performance with large exchange database >>>>>> files >>>>>> >>>>>>> I'm using a mixture of FreeBSD w/ ZFS+snapshots and >>>>>>> rsync to backup all the servers at my day job. This >>>>>>> works pretty good overall but on one server it's not >>>>>>> working so well :) >>>>>>> >>>>>>> We have an Exchange 2003 server with 4 separate mail >>>>>>> store databases. One of them is roughly 900GB the >>>>>>> others are ~200GB, ~160GB, and ~50GB. Rsync seems to >>>>>>> spend a lot of time trying to find the differences in >>>>>>> the files. On the Windows server where rsync is kicked >>>>>>> off there's very little CPU or RAM usage for the rsync >>>>>>> client. On the server rsync (rsyncd, no ssh) is using >>>>>>> around 70-85% of a cpu (well, half a cpu due to hyper >>>>>>> threading). I'm using VSS on the windows server to take >>>>>>> a snapshot and expose it then running rsync from that >>>>>>> to avoid locking issues. >>>>>>> >>>>>>> Is there anything I should check to help narrow down >>>>>>> "problems?" or any settings I should try that could >>>>>>> help speed things up any? >>>>>>> >>>>>>> Below is the final output of the last two rsync runs to >>>>>>> give you an idea. It's taking 30-40+ hours to finish >>>>>>> even though it's only transferring 80-160GB of change. >>>>>>> Right now I'm testing this against a local rsync server >>>>>>> so it should get pretty fast network performance. >>>>>>> Eventually it will be moved to our off-site backup but >>>>>>> that connection is still pretty fast (20 MBbit) and >>>>>>> the backup is only hitting 800-1000 Kbytes/sec. >>>>>>> >>>>>>> >>>>>>> Number of files: 19 Number of files transferred: 6 >>>>>>> Total file size: 1265.74G bytes Total transferred file >>>>>>> size: 1057.06G bytes Literal data: 160.67G bytes >>>>>>> Matched data: 896.39G bytes File list size: 482 File >>>>>>> list generation time: 0.001 seconds File list transfer >>>>>>> time: 0.000 seconds Total bytes sent: 160.71G Total >>>>>>> bytes received: 73.74M >>>>>>> >>>>>>> sent 160.71G bytes received 73.74M bytes 991.84K >>>>>>> bytes/sec total size is 1265.74G speedup is 7.87 >>>>>>> [sender] _exit_cleanup(code=0, >>>>>>> file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, >>>>>>> >>>>>>>line=1052): about to call exit(0)>>>>>>> >>>>>>> real 2833m1.324s user 2225m55.906s sys 45m10.015s >>>>>>> >>>>>>> Number of files: 11 Number of files transferred: 6 >>>>>>> Total file size: 1268.78G bytes Total transferred file >>>>>>> size: 1251.04G bytes Literal data: 83.43G bytes Matched >>>>>>> data: 1167.61G bytes File list size: 216 File list >>>>>>> generation time: 1.360 seconds File list transfer time: >>>>>>> 0.000 seconds Total bytes sent: 83.48G Total bytes >>>>>>> received: 87.25M >>>>>>> >>>>>>> sent 83.48G bytes received 87.25M bytes 836.85K >>>>>>> bytes/sec total size is 1268.78G speedup is 15.18 >>>>>>> >>>>>>> real 1745m5.647s user 1129m14.000s sys 39m58.875s >>>>>>> >>>>>>> Thanks (in advance) for the help :) >>>>>> >>>>>> >>>>>> rsync options I'm using on the client are : -rltihv >>>>>> --progress --stats --inplace --modify-window=1 >>>>>> >>>>>> On the windows client I'm using cygwin + rsync 3.0.9 but >>>>>> I'm going to test 3.1.0 there and see if there's a >>>>>> difference. >>>>>> >>>>>> On the server it's rsync 3.1.0 running rsyncd. >>>>>> >>>>>> Perhaps useful bit of information :). >>>>>> >>>>>> >>>>>> > >>>> -- Please use reply-all for most replies to avoid omitting >>>> the mailing list. To unsubscribe or change options: >>>> https://lists.samba.org/mailman/listinfo/rsync Before >>>> posting, read: >>>> http://www.catb.org/~esr/faqs/smart-questions.html >>> >>> >>> -- Please use reply-all for most replies to avoid omitting the >>> mailing list. To unsubscribe or change options: >>> https://lists.samba.org/mailman/listinfo/rsync Before posting, >>> read: http://www.catb.org/~esr/faqs/smart-questions.html >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlL5TgsACgkQVKC1jlbQAQfNTwCfUV1iYYTujpQLHQxdBD1KFLm8 N9kAoMlhdECrmIT/LwC0tUHgXcYMjDfI =42pY -----END PGP SIGNATURE-----
Maybe Matching Threads
- Rsync performance with large exchange database files
- Problems with dual ldap auth
- Dict issue with PostgreSQL for last_login plugin (duplicate key)
- ODBC to Web Service -- any advice?
- [Fwd: Re:Dovecot] Finally PEBKAC was Failed authentication to a mysql database (bug or PEBKAC?)]