I'm running into a memory problem when trying to switch some web servers
that are running centos4.8/x86_64 to centos5.5/x86_64. The new servers
running centos5.5 are using a decent amount more memory than the
centos4.8 servers.
Here's an example from top:
--- centos 5.5 ---
$ uname -a
Linux ws51 2.6.18-194.17.1.el5 #1 SMP Wed Sep 29 12:50:31 EDT 2010 x86_64 x86_64
x86_64 GNU/Linux
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7799 nobody 16 0 348m 101m 43m S 0.0 2.6 1:10.91 httpd
20285 nobody 15 0 340m 94m 43m S 0.0 2.4 1:11.36 httpd
31734 nobody 15 0 340m 91m 41m S 0.0 2.3 1:13.52 httpd
8904 nobody 15 0 341m 89m 39m S 0.0 2.3 0:35.28 httpd
7353 nobody 15 0 336m 87m 42m S 0.0 2.2 1:21.17 httpd
26097 nobody 15 0 333m 87m 43m S 0.0 2.2 1:28.84 httpd
20765 nobody 15 0 335m 86m 42m S 0.0 2.2 0:48.50 httpd
23299 nobody 15 0 334m 86m 42m S 0.0 2.2 1:13.35 httpd
--- centos 4.8 ---
$ uname -a
Linux ws89 2.6.9-89.0.28.ELsmp #1 SMP Fri Aug 20 16:11:39 EDT 2010 x86_64 x86_64
x86_64 GNU/Linux
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25117 nobody 16 0 244m 57m 41m S 0.0 1.5 0:58.23 httpd
21318 nobody 15 0 240m 53m 41m S 0.0 1.3 0:53.25 httpd
18517 nobody 16 0 239m 51m 41m S 0.0 1.3 0:41.97 httpd
10383 nobody 15 0 238m 50m 40m S 0.0 1.3 0:31.07 httpd
29560 nobody 16 0 239m 49m 39m S 0.0 1.3 0:40.39 httpd
32459 nobody 16 0 238m 49m 40m S 0.0 1.3 0:23.07 httpd
8441 nobody 15 0 238m 48m 39m S 0.0 1.2 0:22.87 httpd
1350 nobody 15 0 238m 48m 39m S 0.0 1.2 0:29.83 httpd
The versions of both apache and php I have installed are the same
between the two releases, so I can't imagine that being the problem.
I've also tested newer versions and it doesn't seem to make a
difference.
I've looked into this quite a bit and below is what I've figured out so
far.
Looking at the output of running 'pmap -x <pid>' shows most of the
libraries in 5.5 are using 2044K or 2048K of memory, but in 4.8 they're
all using 1024K or less. Example:
--- centos 5.5 ---
00002b37c3248000 16 16 0 r-x-- libnss_dns-2.5.so
00002b37c324c000 2044 0 0 ----- libnss_dns-2.5.so
00002b37c344b000 4 4 4 r---- libnss_dns-2.5.so
00002b37c344c000 4 4 4 rw--- libnss_dns-2.5.so
--- centos 4.8 ---
0000002aa4a5e000 16 - - - r-x-- libnss_dns-2.3.4.so
0000002aa4a62000 1020 - - - ----- libnss_dns-2.3.4.so
0000002aa4b61000 4 - - - r---- libnss_dns-2.3.4.so
0000002aa4b62000 4 - - - rw--- libnss_dns-2.3.4.so
This seems to be explained by an increase in ELF_MAXPAGESIZE between the
two releases according to this page:
http://www.greenend.org.uk/rjk/2009/dataseg.html
Another oddity I found that was only in centos 5.5 was a lot of heap
memory being marked as Private_Dirty according to /proc/<pid>/smaps:
2b37c9826000-2b37cc4ca000 rw-p 2b37c9826000 00:00 0 [heap]
Size: 45712 kB
Rss: 45424 kB
Shared_Clean: 0 kB
Shared_Dirty: 1580 kB
Private_Clean: 0 kB
Private_Dirty: 43844 kB
Swap: 0 kB
Pss: 43913 kB
The highest amount of Private_Dirty showing up on 4.8 is just 1628 kB.
Does anyone have any ideas on why 5.5 is using so much memory per httpd
process than it does in 4.8?
I have 100+ machines I want to upgrade, but that'll have to wait until I
figure out why apache uses so much more memory in 5.5...
Thanks in advance,
Bryan
