CentOS - Jun 2015 - Effectiveness of CentOS vm.swappiness

Hi all,

This might not be CentOS related at all. Sorry about that.

I have lots of C6 & C7 machines in use and all of them have the default
swappiness of 60. The problem now is that a lot of those machines do
swap although there is no memory pressure. I'm now thinking about
lowering swappiness to 1. But I'd still like to find out why this
happens. The only common thing between all those machines is that there
are nightly backups done with Bacula. I once came across issues with the
fs-cache bringing Linux to start paging out. Any hints, explanations and
suggestions would be much appreciated.

Cheers, Shorty

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20150604/bb45d147/attachment-0001.sig>

On 04.06.2015 22:18, Markus "Shorty" Uckelmann
wrote:
> Hi all,
> 
> This might not be CentOS related at all. Sorry about that.
> 
> I have lots of C6 & C7 machines in use and all of them have the default
> swappiness of 60. The problem now is that a lot of those machines do
> swap although there is no memory pressure. I'm now thinking about
> lowering swappiness to 1. But I'd still like to find out why this
> happens. The only common thing between all those machines is that there
> are nightly backups done with Bacula. I once came across issues with the
> fs-cache bringing Linux to start paging out. Any hints, explanations and
> suggestions would be much appreciated.
If I'd have to venture a guess then I'd say there are memory pages that
are never touched by any processes and as a result the algorithm has
decided that it's more effective to swap out these pages to disk and use
the freed ram for the page-cache.
Swap usage isn't inherently evil and what you really want to check for
is the si/so columns in the output of the "vmstat" command. If the
system is using swap space but these columns are mostly 0 then that
means memory has been swapped out in the past but there is no actual
swap activity happening right now and there should be no performance
impact. If however these numbers are consistently larger than 0 then
then that means the system is under acute memory pressure and has to
constantly move pages between ram and disk and that will have a large
negative performance impact on the system. This is the moment when swap
usage becomes bad.

Regards,
  Dennis

Am 05.06.2015 um 00:23 schrieb Dennis Jacobfeuerborn:
> If I'd have to venture a guess then I'd say there are memory pages
that
> are never touched by any processes and as a result the algorithm has
> decided that it's more effective to swap out these pages to disk and
use
> the freed ram for the page-cache.
That's my guess too.

[...]
> impact. If however these numbers are consistently larger than 0 then
> then that means the system is under acute memory pressure and has to
> constantly move pages between ram and disk and that will have a large
> negative performance impact on the system. This is the moment when swap
> usage becomes bad.
Gladly I don't have constant paging on all systems. And if there is
paging activity it's very low. AFAIK it's, as you already suggested,
just that some (probably unused) parts are swapped out. But, some of
those parts are the salt-minion, php-fpm or mysqld. All services which
are important for us and which suffer badly from being swapped out. I
already made some tests with swappiness 10 which mildly made it better.
But there still was swap usage. So I tend to set swappiness to 1. Which
I don't like to do, since those default values aren't there for nothing.

Is it possible that this happens because the servers are VMs on an
ESX-server. How could that affect this? How can I further debug this
problem and find out what's the culprit? I will go back to our metrics
and see if I can find any patterns/correlations.

Cheers, Shorty

On Thu, Jun 4, 2015 at 4:18 PM, Markus "Shorty" Uckelmann <shorty
at koeln.de>
wrote:
> I have lots of C6 & C7 machines in use and all of them have the default
> swappiness of 60. The problem now is that a lot of those machines do
> swap although there is no memory pressure. I'm now thinking about
> lowering swappiness to 1. But I'd still like to find out why this
> happens.
>
 Thanks for this thread. I'm actually looking at the same settings for a
different reason.  Most of our environment is VMWare-based and one major
difference between the Linux and Windows clients is how they use
"free"
memory. Linux grabs it for cache ("Free memory is wasted memory.") but
Windows doesn't appear to touch it at all. This means the VMWare hypervisor
can over-commit memory.

Am 04.06.2015 um 22:18 schrieb Markus "Shorty"
Uckelmann:
> Hi all,
Thanks for all your help!

I just found a few additional things one can or should do when 
investigating swap-related issues:

* dmesg - always do that!
* Look for a RAM-disk. These things are kernel memory. So they don't 
show up in "smem" et.al.
* Take a look at some of swaps contents:
strings -f /dev/sda3 | shuf -n  100000  | less


Cheers, Shorty