Are you running multiple rsyncs in parallel? I don't see how a single
rsync can use up 4 CPUs when rsync is only 1 or 2 processes depending on
whether it is sending or receiving, respectively. If you are running
multiple rsyncs, cut down on the number you're doing in parallel.
Rsync does use up a bit of memory for every file it visits in a single
run. A redesign to avoid that has long been under discusion but it
hasn't been implemented.
- Dave Dykstra
On Mon, Jun 17, 2002 at 09:24:35AM -0700, Sandy Ganz
wrote:> Hello -
>
> I'm running Redhat 7.1 Rsync 2.4.6 and am having some problems with the
> CPU's getting swammped. What we have is basically an 3 boxes, 1 machine
(the
> factory) is a where applications store the images that eventually are used
> for the site, then the other 2 are the webservers that the have copy of the
> images. The person who set the machines up used rsync to move any new
images
> from the 'factory box' to both of the webserver boxes. The amount
of images
> are huge on all boxes (close to a million). Typically a few thousands
images
> is all that rsync must move. The directory structure where the images live
> are broken down into a shallow and somewhat wide tree with a few thousand
> images in each directory
>
> The problem...
>
> When rsync seems to to build the list of files, seems OK, uses about 20meg
> of ram (not swaping either). Then when the files start to be moved (all
> around 2-4k) the cpu's (4x 500 xeon) all run 100% and kill any Http
trafic
> that need also to be running on the box. This would be ok if on the Source
> factory box, but not on the web servers. The interesting thing is that is
> does not matter if I run Rsync on the Factory or the Webservers it only
ever
> pegs the CPU's on the web servers.
>
> I tried the throttle on bandwidth [at 256], was ng, as that might be for
> file transfers of a larger size. This is lots of small .jpg images.
>
> Any ideas on how to keep rsync from using all the cpu on the webservers? I
> got tossed this problem as I was a warm body, but am not a linux master so
> might be I got stuck with some bad use of rsync.
>
> Also it seems odd that both the source and dest machings use a huge amount
> of ram, I would expect only one to need to do the file compair. Does this
> sound wrong?
>
> Thanks for any help you can offer
>
> Sandy
>
>
>
> --
> To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html