I'm trying to sync up 54 million files. I can break it down into different applications, but I still have to accomplish 17 million files in one "chunk" if possible. I'm using a fast (v440) cpu system with 32G ram. Originally I was running out of memory (seemed to increase like crazy with --delete on). I've upgraded to rsync 2.6.6 and the memory seems to be more stable now, but it's still not using much CPU to perform it's task, I have CPU to spare! I'm trying to sync from one NFS mount to another NFS mount (Netapp). Is there anyway I can force rsync to use *more* resources? My Netapp doesn't seem to be pounded and my V440 is only using 2%-8% of cpu, 27G of real memory free. Am I doing something wrong? It's going on 5 hours now. It's "considered" all the files and printed out a few it deleted, but that this point I'm not sure where it is at. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Hi Jerry, I reply offlist because I'm everything but an expert, so you might get a better answer from someone else. Anyway, perhaps it would be good if you post to the list what you're already doing. On Tuesday, 14. February 2006 03:31, Jerry wrote:> I'm trying to sync up 54 million files. I can break > it down into different applications, but I still have > to accomplish 17 million files in one "chunk" if > possible.Uh oh :-)> I'm using a fast (v440) cpu system with 32G ram. > Originally I was running out of memory (seemed to > increase like crazy with --delete on). I've upgraded > to rsync 2.6.6 and the memory seems to be more stable > now, but it's still not using much CPU to perform it's > task, I have CPU to spare! I'm trying to sync from > one NFS mount to another NFS mount (Netapp).Could you do that on one of the machines you have NFS-mounted now? Although I'm not sure at all, I *believe* that rsync will consider this situation as a local-to-local copy and may optimise for that, but since there are 2 copies over-a-network (one from 'source' to 'here' (where the CPU is that does all the work) and one from 'here' to 'target'), this could be quite inefficient. If you run the rsync either on the source machine or the target machine, it MIGHT become about twice as fast, because there is only one copy now. Further, rsync might recognise the situation as remote-to-local or local-to-remote copying and optimise for low network load.> Is there anyway I can force rsync to use *more* > resources? My Netapp doesn't seem to be pounded and > my V440 is only using 2%-8% of cpu, 27G of real memory > free. Am I doing something wrong? It's going on 5 > hours now. It's "considered" all the files and > printed out a few it deleted, but that this point I'm > not sure where it is at.This block is something for the real experts. Only one thing: You could perhaps use on-the-fly compression to use more CPU and lower the network load. Good Luck & Cheers Dirk
I think rsync over two nfs mounted volumes is an ongoing 'problem', partly (mainly?) because people forget that these are 'local' files and lots of the neato rsync speedups don't do anything or just get ignored. If there is no way for you to run a separate rsync on local and remote machines, your file transfer might be better accomplished with some other method according to the old thread mentioned below I am sure there are other technical issues as well ... this has been discussed before, see this thread for example. http://lists.samba.org/archive/rsync/2002-October/003970.html Disclaimer, I am another user not a developer, if any developers know about newer information I am sure they will correct me.
Because of the inefficiencies of using NFS to NFS rsync, would you not be better using a native utility from filer to file, such as ndmp copy?>>> Jerry <juanino@yahoo.com> 02/13/06 09:31PM >>>I'm trying to sync up 54 million files. I can break it down into different applications, but I still have to accomplish 17 million files in one "chunk" if possible. I'm using a fast (v440) cpu system with 32G ram. Originally I was running out of memory (seemed to increase like crazy with --delete on). I've upgraded to rsync 2.6.6 and the memory seems to be more stable now, but it's still not using much CPU to perform it's task, I have CPU to spare! I'm trying to sync from one NFS mount to another NFS mount (Netapp). Is there anyway I can force rsync to use *more* resources? My Netapp doesn't seem to be pounded and my V440 is only using 2%-8% of cpu, 27G of real memory free. Am I doing something wrong? It's going on 5 hours now. It's "considered" all the files and printed out a few it deleted, but that this point I'm not sure where it is at. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How about a shell script? Scan the directories and start "x" instances of rsync at a time, when one finished, have the script fire up another, until the list of directories is complete......? Gian> >The design is outside of my control. It's not a >regular replication it's a migration from one netapp >nfs appliance to another. (And no, I can't use the >wonderful block-level replication they provide due to >some other complex reasons). > >I'm wondering if there is a way to tell rsync, go >ahead, take each dir off the top level and fire up an >independent rsync to do this. It seems it's not using >all my cpu, memory, or bandwidth (on either side) to >accomplish the task. I'm not sure where I'm being >throttled, except maybe the NFS client on the Solaris >box not being very efficient (Solaris 10, so it >*should* be). > >--- Frank Hamersley <terabite@bigpond.com> wrote: > > > >>>-----Original Message----- >>>From: >>> >>> >>rsync-bounces+terabite=bigpond.com@lists.samba.org >> >> >>>Behalf Of Dirk Schenkewitz >>>Sent: Tuesday, 14 February 2006 10:32 PM >>>To: rsync@lists.samba.org >>>Subject: Re: sync 54 million files, tuning rsync? >>> >>> >>(offlist) >> >> >>>Hi Jerry, >>> >>>I reply offlist because I'm everything but an >>> >>> >>expert, so >> >> >>>you might get a better answer from someone else. >>>Anyway, perhaps it would be good if you post to >>> >>> >>the list >> >> >>>what you're already doing. >>> >>>On Tuesday, 14. February 2006 03:31, Jerry wrote: >>> >>> >>>>I'm trying to sync up 54 million files. I can >>>> >>>> >>break >> >> >>>>it down into different applications, but I still >>>> >>>> >>have >> >> >>>>to accomplish 17 million files in one "chunk" if >>>>possible. >>>> >>>> >>[snip] >>Jerry, >> >><OutsideTheSquare> >>Given you have 54 million entries have you ever >>considered using a genuine DBMS (with replication) >>rather than pounding the filesystem to death? >></OutsideTheSquare> >> >>Cheers, Frank. >> >>-- >>To unsubscribe or change options: >>https://lists.samba.org/mailman/listinfo/rsync >>Before posting, read: >>http://www.catb.org/~esr/faqs/smart-questions.html >> >> >> > > >__________________________________________________ >Do You Yahoo!? >Tired of spam? Yahoo! Mail has the best spam protection around >http://mail.yahoo.com > > > >------------------------------------------------------------------------ > >_______________________________________ >Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > >rsync mailing list >rsync@lists.samba.org >https://lists.samba.org/mailman/listinfo/rsync > >