CentOS - Jul 2009 - out of memory

I am getting this message quite often lately.

Centos 5.3, AMD dual core 5050e system x64

1 GIG ram, 4 GIG swap

I dont have that much running that the kernel should be cutting out my
processes.

Any thoughts?

Jerry

------------


emTotal:       766264 kB
MemFree:        583984 kB
Buffers:          9572 kB
Cached:          31004 kB
SwapCached:      36280 kB
Active:          20240 kB
Inactive:        59780 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       766264 kB
LowFree:        583984 kB
SwapTotal:     4096564 kB
SwapFree:      3932136 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:       13184 kB
Mapped:          16340 kB
Slab:            22428 kB
PageTables:      13980 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   4479696 kB
Committed_AS:   442124 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    280856 kB
VmallocChunk: 34359457495 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB


096kB = 3424kB
Node 0 Normal: empty
Node 0 HighMem: empty
46 pagecache pages
Swap cache: add 8934611, delete 8934611, find 7869746/7929974, race 11+11
Free swap  = 0kB
Total swap = 4096564kB
Free swap:            0kB
196064 pages of RAM
13511 reserved pages
5948 pages shared
0 pages swap cached
kthread invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0

Call Trace:
 [<ffffffff800c3a6a>] out_of_memory+0x8e/0x2f5
 [<ffffffff8000f2eb>] __alloc_pages+0x245/0x2ce
 [<ffffffff8002303f>] alloc_page_interleave+0x3d/0x74
 [<ffffffff8003c061>] __get_free_pages+0xe/0x71
 [<ffffffff8001eb82>] copy_process+0xc6/0x15b8
 [<ffffffff8009b8bc>] alloc_pid+0x1ee/0x28a
 [<ffffffff80030cdb>] do_fork+0x69/0x1c1
 [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8005df3d>] kernel_thread+0x81/0xeb
 [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032282>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11
 [<ffffffff8009d9a9>] keventd_create_kthread+0x1d/0xc4
 [<ffffffff8004d159>] run_workqueue+0x94/0xe4
 [<ffffffff800499da>] worker_thread+0x0/0x122
 [<ffffffff80049aca>] worker_thread+0xf0/0x122
 [<ffffffff8008a4b3>] default_wake_function+0x0/0xe
 [<ffffffff80032380>] kthread+0xfe/0x132
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff80032282>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11

Mem-info:
Node 0 DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
Node 0 DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:146
cpu 0 cold: high 62, batch 15 used:47
cpu 1 hot: high 186, batch 31 used:18
cpu 1 cold: high 62, batch 15 used:55
Node 0 Normal per-cpu: empty
Node 0 HighMem per-cpu: empty
Free pages:        5652kB (0kB HighMem)
Active:78394 inactive:78664 dirty:0 writeback:0 unstable:0 free:1413 slab:5343
mapped-file:6627 mapped-anon:157109 pagetables:5859
Node 0 DMA free:2160kB min:48kB low:60kB high:72kB active:0kB inactive:0kB
present:10724kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 739 739 739
Node 0 DMA32 free:3492kB min:3452kB low:4312kB high:5176kB active:313576kB
inactive:314656kB present:757376kB pages_scanned:1226904 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 4*8kB 2*16kB 5*32kB 4*64kB 1*128kB 2*256kB 0*512kB 1*1024kB
0*2048kB 0*4096kB = 2160kB
Node 0 DMA32: 19*4kB 11*8kB 0*16kB 0*32kB 8*64kB 2*128kB 0*256kB 1*512kB
0*1024kB 1*2048kB 0*4096kB = 3492kB
Node 0 Normal: empty
Node 0 HighMem: empty
45 pagecache pages
Swap cache: add 8934620, delete 8934620, find 7869746/7929974, race 11+11
Free swap  = 0kB
Total swap = 4096564kB
Free swap:            0kB
196064 pages of RAM
13511 reserved pages
4950 pages shared
0 pages swap cached

On Wed, Jul 29, 2009 at 4:30 PM, Jerry Geis<geisj at pagestation.com>
wrote:
> I am getting this message quite often lately.
>
> Centos 5.3, AMD dual core 5050e system x64
>
> 1 GIG ram, 4 GIG swap
>
Well that is a lot of swap for a server.. if something really starts
using that much swap its going to most likely end up in a bad race
where you are swapping out more stuff than you are getting in.  For
most large memory systems my rule of thumb is no more than 1:1 on
swap. Adding to 1 to 3 more GB of memory can fix some of these issues
sometimes because you have more room to stuff things in when trying to
swap stuff in.
> Free swap ?= 0kB
> Total swap = 4096564kB
> Free swap: ? ? ? ? ? ?0kB
Something is eating up swap. Not sure what it is from the message.. so
you are going to need to look in using some sort of script to track
whats going on and see if you can find out what is doing it.



-- 
Stephen J Smoogen.

Ah, but a man's reach should exceed his grasp. Or what's a heaven for?
-- Robert Browning

Jerry Geis wrote on Wed, 29 Jul 2009 18:30:05 -0400:
> Any thoughts?
Doesn't top help in finding out what's eating your RAM?

Kai

-- 
Kai Sch?tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com