CentOS - Jun 2015 - Effectiveness of CentOS vm.swappiness

Am 05.06.2015 um 00:23 schrieb Dennis Jacobfeuerborn:
> If I'd have to venture a guess then I'd say there are memory pages
that
> are never touched by any processes and as a result the algorithm has
> decided that it's more effective to swap out these pages to disk and
use
> the freed ram for the page-cache.
That's my guess too.

[...]
> impact. If however these numbers are consistently larger than 0 then
> then that means the system is under acute memory pressure and has to
> constantly move pages between ram and disk and that will have a large
> negative performance impact on the system. This is the moment when swap
> usage becomes bad.
Gladly I don't have constant paging on all systems. And if there is
paging activity it's very low. AFAIK it's, as you already suggested,
just that some (probably unused) parts are swapped out. But, some of
those parts are the salt-minion, php-fpm or mysqld. All services which
are important for us and which suffer badly from being swapped out. I
already made some tests with swappiness 10 which mildly made it better.
But there still was swap usage. So I tend to set swappiness to 1. Which
I don't like to do, since those default values aren't there for nothing.

Is it possible that this happens because the servers are VMs on an
ESX-server. How could that affect this? How can I further debug this
problem and find out what's the culprit? I will go back to our metrics
and see if I can find any patterns/correlations.

Cheers, Shorty

On Fri, Jun 05, 2015 at 12:29:04PM +0200, Markus "Shorty" Uckelmann
wrote:
> How can I further debug this
> problem and find out what's the culprit?
It's working as designed.

Linux does not treat various kinds of memory pages differently. If you
want a daemon to be fully in core, call mlockall(). Here's one way to
do that without changing the daemon's source:

http://superuser.com/questions/196725/how-to-mlock-all-pages-of-a-process-tree-from-console

(I've always only done this with my own code explicitly calling mlock)

If you don't explicitly lock things into memory, file I/O can and will
cause idle pages to get pushed out. It happens less often if you
manipulate swappines.

-- greg

On 06/05/2015 03:29 AM, Markus "Shorty" Uckelmann
wrote:
> some (probably unused) parts are swapped out. But, some of
> those parts are the salt-minion, php-fpm or mysqld. All services which
> are important for us and which suffer badly from being swapped out.
Those two things can't really both be true.  If the pages swapped out 
are unused, then the application won't suffer as a result.

On Fri, Jun 05, 2015 at 09:33:11AM -0700, Gordon Messmer
wrote:
> On 06/05/2015 03:29 AM, Markus "Shorty" Uckelmann wrote:
> >some (probably unused) parts are swapped out. But, some of
> >those parts are the salt-minion, php-fpm or mysqld. All services which
> >are important for us and which suffer badly from being swapped out.
> 
> Those two things can't really both be true.  If the pages swapped
> out are unused, then the application won't suffer as a result.
No.

Let's say the application only uses the page once per hour. If there
is also I/O going on, then it's easy to see that the kernel could
decide to page the page out after 50 minutes, leaving the application
having to page it back in 10 minutes later.

-- greg

Am 05.06.2015 um 18:33 schrieb Gordon Messmer:
> On 06/05/2015 03:29 AM, Markus "Shorty" Uckelmann wrote:
>> some (probably unused) parts are swapped out. But, some of
>> those parts are the salt-minion, php-fpm or mysqld. All services which
>> are important for us and which suffer badly from being swapped out.
>
> Those two things can't really both be true.  If the pages swapped out
> are unused, then the application won't suffer as a result.
Why not? If you have an application which sees action only every 12 to 
24 hours,I think this can happen. Our salt-minion would be a candidate 
for this. Allthough we constantly check if it's alive, we only do once 
or twice a day something "heavy" like a deployment. And very often we 
have to run thos deployments twice, because the first time we get a lot 
of timeouts. Sure, it might be the software itself. But I think it could 
be possible that it is because of swapped out pages.

I can't be sure about this. That's why I want to find out what and why 
things are happening. But first I need to find the right tools to do this ;)

Cheers, Shorty

Am 05.06.2015 um 17:40 schrieb Greg Lindahl:
> On Fri, Jun 05, 2015 at 12:29:04PM +0200, Markus "Shorty"
Uckelmann wrote:
>
>> How can I further debug this
>> problem and find out what's the culprit?
>
> It's working as designed.
Sadly. It is just my first time I see this behaviour to this extent/on 
so many servers. So you can say that I'm kind of a newbie in swapping ;)
> If you don't explicitly lock things into memory, file I/O can and will
> cause idle pages to get pushed out. It happens less often if you
> manipulate swappines.
So, is a swappiness value of 60 not recommended for servers? I worked 
with hundreds of servers (swappiness 60) on a social platform and 
swapping very rarely happened and then only on databases (which had 
swappiness set to 0). The only two differences (that I can see) to my 
current servers are that I used Debian and there was no "extra" I/O
from
backups.

I might be overstating the swapping thing. That's what I'm trying to 
find out.

Cheers, Shorty

On Fri, Jun 05, 2015 at 08:40:27AM -0700, Greg Lindahl
wrote:
> Linux does not treat various kinds of memory pages differently. If you
> want a daemon to be fully in core, call mlockall(). Here's one way to
> do that without changing the daemon's source:
Another way to do this is to put the services into a named CGroup, and
set memory.swappiness=1 for that cgroup.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html

Not necessarily as effective as mlock() but you might want to set some
of the other cgroup features as well.

-- 
Jonathan Billings <billings at negate.org>