CentOS - Jun 2006 - Swap memory: I can't reconcile this stuff.

I need to look more into it, but before I start the long and arduous
"googling my life away" process, I figured someone might know the
answer. I've read the man pages several times and they didn't change!
:-(

As normal, while looking at one thing, something else bites my butt. I
tuned on the swap field in top and sort on it. Here's an edited snippet
of the results.

Mem:    775708k total,   764752k used,    10956k free,    60780k buffers
Swap:  1572856k total,      160k used,  1572696k free,   377324k cached

  PID  VIRT  RES  SHR %MEM SWAP COMMAND
24729  127m  32m  15m  4.3  94m evolution
 3409 97220 5268 4304  0.7  89m evolution-data-
 2851  115m  36m 7120  4.8  79m X
10937  109m  45m  14m  6.0  63m firefox-bin
 3417 63076 7876 6756  1.0  53m evolution-alarm
 3363 40332 7284 6228  0.9  32m eggcups
24745 37480 8176 6876  1.1  28m evolution-excha
 3736 53272  29m 8660  3.9  22m gnome-terminal
 3361 44404  21m  10m  2.9  21m nautilus
 3357 39868  21m  10m  2.8  17m gnome-panel
 4096 25460 7600 5960  1.0  17m gkrellm
 3373 20488 3492 2944  0.5  16m gnome-vfs-daemo
 3367 43608  26m  10m  3.4  16m rhn-applet-gui
 3359 19904 6128 5316  0.8  13m gnome-volume-ma
 3387 20904 8068 6696  1.0  12m clock-applet
 3389 19456 6648 5696  0.9  12m notification-ar
 3316 19080 7140 5960  0.9  11m gnome-settings-
 3385 22304  10m 7576  1.4  11m mixer_applet2
 3244 21508 9960 6868  1.3  11m gnome-session
 4144 22476  10m 7456  1.4  11m wnck-applet
 2587 12412 2364 1940  0.3 9.8m gdm-binary
 2846 13220 3340 2728  0.4 9880 gdm-binary
 3365 13812 4532 3920  0.6 9280 pam-panel-icon
 3355 14768 7524 5984  1.0 7244 metacity
 7182 10328 3436 2280  0.4 6892 sendmail
18501 11080 4248 1912  0.5 6832 cupsd

Note that the summary line says 160k of swap is used. The man pages say
the summary and the details under "SWAP" are both reported in
"k". No
mention of "m" is made, I presume that it means "megabytes"?

If so, I can't reconcile what I'm seeing. Free seems to support the
summary lines.

             total       used       free...
Mem:        775708     764772      10936...
-/+ buffers/cache:     326584     449124
Swap:      1572856        160    1572696

Cat of /proc/meminfo also seems to support the summary lines.

SwapTotal:     1572856 kB
SwapFree:      1572696 kB

Now, if I treat all those numbers ending in "m" as megabytes, it
doesn't
take long to see that I've been lied to somewhere along the way. Or
alternatively, I'm dense and "Just Don't Get It" (TM).

Any help?

-- 
Bill
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.centos.org/pipermail/centos/attachments/20060605/d5f48a54/attachment-0002.sig>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, Jun 05, 2006 at 06:45:10PM -0400, William L. Maltby
wrote:
> I need to look more into it, but before I start the long and arduous
> "googling my life away" process, I figured someone might know the
> answer. I've read the man pages several times and they didn't
change!
> :-(
> 
> As normal, while looking at one thing, something else bites my butt. I
> tuned on the swap field in top and sort on it. Here's an edited snippet
> of the results.
> 
> Mem:    775708k total,   764752k used,    10956k free,    60780k buffers
> Swap:  1572856k total,      160k used,  1572696k free,   377324k cached
> 
>   PID  VIRT  RES  SHR %MEM SWAP COMMAND
> 24729  127m  32m  15m  4.3  94m evolution
>  3409 97220 5268 4304  0.7  89m evolution-data-
>  2851  115m  36m 7120  4.8  79m X
>  3359 19904 6128 5316  0.8  13m gnome-volume-ma
(...)
>  3355 14768 7524 5984  1.0 7244 metacity
>  7182 10328 3436 2280  0.4 6892 sendmail
> 18501 11080 4248 1912  0.5 6832 cupsd
> 
> Note that the summary line says 160k of swap is used. The man pages say
> the summary and the details under "SWAP" are both reported in
"k". No
> mention of "m" is made, I presume that it means
"megabytes"?
> 
> If so, I can't reconcile what I'm seeing. Free seems to support the
> summary lines.
> 
>              total       used       free...
> Mem:        775708     764772      10936...
> -/+ buffers/cache:     326584     449124
> Swap:      1572856        160    1572696
> 
> Cat of /proc/meminfo also seems to support the summary lines.
> 
> SwapTotal:     1572856 kB
> SwapFree:      1572696 kB
> 
> Now, if I treat all those numbers ending in "m" as megabytes, it
doesn't
> take long to see that I've been lied to somewhere along the way. Or
> alternatively, I'm dense and "Just Don't Get It" (TM).
> 
> Any help?
I can see two possible explanations for this. Maybe both in conjunction.

One is that you get getting multiple entries for the same processes,
but different threads. That used to be the way of it up until .. humm,
not sure ... 2.4, I guess. Not sure exactly how it works these days. I would
have to check.

The other is the overcommit kernel feature. It is possible the kernel
is overcommiting memory, and thus showing more than it really in
use.

One last thing possible (just thought about it) is that top is adding
more than one namespace to those totals. Maybe shared memory (/dev/shm ?).
Or any other possible namespace.

I agree it does seem odd, and I have seen this kind of stuff happening
before. Once I started hunting it down, and found the reason for it.
It was some time ago (2.2 ? 2.4 ? Not sure), so I'm reasonably sure
it is not the same reason these days. But I hope I gave you are
least some pointers for where to start looking.


- -- 
Rodrigo Barbosa <rodrigob at suespammers.org>
"Quid quid Latine dictum sit, altum viditur"
"Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFEhLfppdyWzQ5b5ckRArwRAKCmZiNBRKpo4H+w9yz7iOFxPDXpWQCeMUUB
xl9CoW3fjVtw3oYRSFFWfQU=BOYa
-----END PGP SIGNATURE-----

William L. Maltby wrote on Mon, 05 Jun 2006 18:45:10 -0400:
> 24729  127m  32m  15m  4.3  94m evolution
It looks to me like top is using a different definition of swap. If you 
compare you see that SWAP shows the difference between VIRT and RES almost 
exactly. However, if there is no or almost no swap usage how can there be 
a difference? Only explanation: it shows what *could* be swapped 
immediately without affecting current activity. 
Have you tried making it heavily swap and then checked if you see a 
difference (whichever)? Theoretically, it shouldn't make a difference as 
it doesn't seem to differentiate between "swapped" and
"really swapped".

Kai

-- 
Kai Sch?tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

On Mon, 2006-06-05 at 18:45 -0400, William L. Maltby
wrote:
> I need to look more into it, but before I start the long and arduous
> "googling my life away" process, I figured someone might know the
> answer. I've read the man pages several times and they didn't
change!
> :-(
> 
> As normal, while looking at one thing, something else bites my butt. I
> tuned on the swap field in top and sort on it. Here's an edited snippet
> of the results.
> 
> Mem:    775708k total,   764752k used,    10956k free,    60780k buffers
> Swap:  1572856k total,      160k used,  1572696k free,   377324k cached
> 
>   PID  VIRT  RES  SHR %MEM SWAP COMMAND
> 24729  127m  32m  15m  4.3  94m evolution
>  3409 97220 5268 4304  0.7  89m evolution-data-
>  2851  115m  36m 7120  4.8  79m X
[...]
> Note that the summary line says 160k of swap is used. The man pages say
> the summary and the details under "SWAP" are both reported in
"k". No
> mention of "m" is made, I presume that it means
"megabytes"?
> 
> If so, I can't reconcile what I'm seeing. Free seems to support the
> summary lines.
> 
>              total       used       free...
> Mem:        775708     764772      10936...
> -/+ buffers/cache:     326584     449124
> Swap:      1572856        160    1572696
> 
> Cat of /proc/meminfo also seems to support the summary lines.
> 
> SwapTotal:     1572856 kB
> SwapFree:      1572696 kB
> 
> Now, if I treat all those numbers ending in "m" as megabytes, it
doesn't
> take long to see that I've been lied to somewhere along the way. Or
> alternatively, I'm dense and "Just Don't Get It" (TM).
Executables aren't copied into swap - virtual memory is allocated but
they page in on demand directly from the binary file.  So there really
are megs that could swap in if other portions of those programs are
executed even though it isn't in the swap partition you designated.

-- 
  Les Mikesell
    lesmikesell at gmail.com

William L. Maltby wrote:
>I need to look more into it, but before I start the long and arduous
>"googling my life away" process, I figured someone might know the
>answer. I've read the man pages several times and they didn't
change!
>:-(
>
>As normal, while looking at one thing, something else bites my butt. I
>tuned on the swap field in top and sort on it. Here's an edited snippet
>of the results.
>
>Mem:    775708k total,   764752k used,    10956k free,    60780k buffers
>Swap:  1572856k total,      160k used,  1572696k free,   377324k cached
>
>  PID  VIRT  RES  SHR %MEM SWAP COMMAND
>24729  127m  32m  15m  4.3  94m evolution
> 3409 97220 5268 4304  0.7  89m evolution-data-
> 2851  115m  36m 7120  4.8  79m X
>10937  109m  45m  14m  6.0  63m firefox-bin
> 3417 63076 7876 6756  1.0  53m evolution-alarm
> 3363 40332 7284 6228  0.9  32m eggcups
>24745 37480 8176 6876  1.1  28m evolution-excha
> 3736 53272  29m 8660  3.9  22m gnome-terminal
> 3361 44404  21m  10m  2.9  21m nautilus
> 3357 39868  21m  10m  2.8  17m gnome-panel
> 4096 25460 7600 5960  1.0  17m gkrellm
> 3373 20488 3492 2944  0.5  16m gnome-vfs-daemo
> 3367 43608  26m  10m  3.4  16m rhn-applet-gui
> 3359 19904 6128 5316  0.8  13m gnome-volume-ma
> 3387 20904 8068 6696  1.0  12m clock-applet
> 3389 19456 6648 5696  0.9  12m notification-ar
> 3316 19080 7140 5960  0.9  11m gnome-settings-
> 3385 22304  10m 7576  1.4  11m mixer_applet2
> 3244 21508 9960 6868  1.3  11m gnome-session
> 4144 22476  10m 7456  1.4  11m wnck-applet
> 2587 12412 2364 1940  0.3 9.8m gdm-binary
> 2846 13220 3340 2728  0.4 9880 gdm-binary
> 3365 13812 4532 3920  0.6 9280 pam-panel-icon
> 3355 14768 7524 5984  1.0 7244 metacity
> 7182 10328 3436 2280  0.4 6892 sendmail
>18501 11080 4248 1912  0.5 6832 cupsd
>
>Note that the summary line says 160k of swap is used. The man pages say
>the summary and the details under "SWAP" are both reported in
"k". No
>mention of "m" is made, I presume that it means
"megabytes"?
>
>If so, I can't reconcile what I'm seeing. Free seems to support the
>summary lines.
>
>             total       used       free...
>Mem:        775708     764772      10936...
>-/+ buffers/cache:     326584     449124
>Swap:      1572856        160    1572696
>
>Cat of /proc/meminfo also seems to support the summary lines.
>
>SwapTotal:     1572856 kB
>SwapFree:      1572696 kB
>
>Now, if I treat all those numbers ending in "m" as megabytes, it
doesn't
>take long to see that I've been lied to somewhere along the way. Or
>alternatively, I'm dense and "Just Don't Get It" (TM).
>
>Any help?
>  
>
Looked at the same top info, and I don't understand what I see either.  
With processes running, I'm seeing close to 500mb of stuff swapped out, 
but according to free, only the 2.2 mb is swapped.  There's gotta be 
more to this than meets the eye, or something is lying about the swap.  
The system monitor also shows only 2.2mb of swap in use, so where is top 
getting this 500+ mb of swap data from?