Hi,
Im running a Dell 6450 w/ 4GB of memory, 2 2GB Swap Partitions. Redhat 7.3
kernel 2.4.18 (kernel.org). It runs our images for our site. We use rsync to
sync images from a main image box. It runs fine for maybe 4 days untill all
memory is in cache, and like 7MB of physical memory is free. The box will
hardly do anything, and cache will just keep going up and up untill no more
memory is free. Then when I run rsync (a memory hog) at night, kswapd and
rsync will compete with each other and send the cpu load through the roof.
Funny thing is the box won't really use swap at all. Then while the box is
rsyncing I cant even do simple things like SSH to the box. Why did older
kernels (2.2 kernels) work a whole lot better. Did older kernels swap a lot
better? Is there anything I can adjust in 2.4.18. Can I flush cache out
manually some how to return that memory back for use, for applications like
rsync. If I could do that my problems would be solved, since rsync runs
great with 1 GB of memory free. ANY HELP Would be much appreciative.
total used free shared buffers cached
Mem: 3997300 3918388 78912 0 472364 3056796
-/+ buffers/cache: 389228 3608072
Swap: 4192880 2460 4190420
10:21am up 6 days, 19:45, 2 users, load average: 0.04, 0.03, 0.00
79 processes: 78 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.1% user, 0.3% system, 0.0% nice, 99.1% idle
CPU1 states: 0.0% user, 0.2% system, 0.0% nice, 99.3% idle
CPU2 states: 0.0% user, 0.1% system, 0.0% nice, 99.4% idle
CPU3 states: 0.2% user, 0.0% system, 0.0% nice, 99.3% idle
Mem: 3997300K av, 3919908K used, 77392K free, 0K shrd, 472680K
buff
Swap: 4192880K av, 2460K used, 4190420K free 3057584K
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
7826 root 16 0 1040 1040 832 R 0.5 0.0 0:00 top
2293 root 9 0 1736 1684 1400 S 0.1 0.0 0:00 sshd
5776 nobody 10 0 1124 1108 972 S 0.1 0.0 0:00 httpd
6385 nobody 11 0 1120 1104 968 S 0.1 0.0 0:00 httpd
6707 nobody 10 0 1120 1108 976 S 0.1 0.0 0:00 httpd
6919 nobody 9 0 1116 1100 964 S 0.1 0.0 0:00 httpd
1 root 10 0 472 428 412 S 0.0 0.0 14:45 init
2 root 9 0 0 0 0 SW 0.0 0.0 50:40 keventd
3 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0
4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1
5 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU2
6 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU3
7 root 9 0 0 0 0 SW 0.0 0.0 305:31 kswapd
8 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
9 root 9 0 0 0 0 SW 0.0 0.0 83:16 kupdated
10 root 9 0 0 0 0 SW 0.0 0.0 13:08 kjournald
122 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
123 root 9 0 0 0 0 SW 0.0 0.0 2:26 kreiserfsd
124 root 9 0 0 0 0 SW 0.0 0.0 1:05 kjournald
125 root 9 0 0 0 0 SW 0.0 0.0 4:50 kjournald
126 root 9 0 0 0 0 SW 0.0 0.0 4:29 kjournald
537 root 9 0 604 536 500 S 0.0 0.0 11:27 syslogd
542 root 9 0 444 428 428 S 0.0 0.0 0:00 klogd
654 root 9 0 1024 912 828 S 0.0 0.0 13:34 sshd
On Wed, Jul 10, 2002 at 01:50:00PM -0700, Rick Parada wrote:> Hi, > > Im running a Dell 6450 w/ 4GB of memory, 2 2GB Swap Partitions. Redhat 7.3 > kernel 2.4.18 (kernel.org). It runs our images for our site. We use rsync to > sync images from a main image box. It runs fine for maybe 4 days untill all > memory is in cache, and like 7MB of physical memory is free. The box will > hardly do anything, and cache will just keep going up and up untill no more > memory is free. Then when I run rsync (a memory hog) at night, kswapd and > rsync will compete with each other and send the cpu load through the roof. > Funny thing is the box won't really use swap at all. Then while the box is > rsyncing I cant even do simple things like SSH to the box. Why did older > kernels (2.2 kernels) work a whole lot better. Did older kernels swap a lot > better? Is there anything I can adjust in 2.4.18. Can I flush cache out > manually some how to return that memory back for use, for applications like > rsync. If I could do that my problems would be solved, since rsync runs > great with 1 GB of memory free. ANY HELP Would be much appreciative. > > total used free shared buffers cached > Mem: 3997300 3918388 78912 0 472364 3056796 > -/+ buffers/cache: 389228 3608072 > Swap: 4192880 2460 4190420You aren't out of memory. And it probably isn't kswapd but bounce buffering that is killing you. I'd bet that if you downgraded to 2GB you'd get snappier response, at least after 4 days. 4GB is fine as long as you aren't doing lots of io. The fact that it takes 4 days to get to this state, and that you aren't paging says to me that you really aren't using your 4GB except for pagecache. As for the 2.2 - 2.4 comparison. These are completely different beasts. 2.4 has major MM changes that are still settling. .From what i can tell the fix is to either upgrade your kernel and turn on CONFIG_HIGHIO or only run with 2GB memory. I don't use RH but it is possible that the HIGHIO patch may have been applied already and you can turn it on. Extracted from linux-kernel:> Subject: Re: Terrible VM in 2.4.11+? > Date: Wed, 10 Jul 2002 10:49:04 +0200 > From: Jens Axboe <axboe@suse.de> > Cc: linux-kernel@vger.kernel.org, > Status: RO > > On Wed, Jul 10 2002, Thomas Tonino wrote: > > > > I run a 2 cpu server with 16 disks and around 5 > > megabytes of writes a second. With plain 2.4.18 (using > > the feral.com qlogic driver) and 2GB ram, this seemed > > okay. Upgrading to 4GB ram slowed the system down, and > > normal shell commands became quite unresponsive with > > 4GB. > > > > So we built a second server, with 2.4.19-pre9-aa2 using > > the qlogic driver in the kernel. That driver needs > > patching, as it will otherwise get stuck in a 'no handle > > slots' condition. Used a patch that I posted to > > linux-scsi a while ago. > > That's probably not just a mm issue, if you use stock > 2.4.18 with 4GB ram you will spend oodles of time bounce > buffering i/o. 2.4.19-pre9-aa2 includes the block-highmem > stuff, which enables direct-to-highmem i/o, if you enabled > the CONFIG_HIGHIO option.-- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt