Marco Fioretti
2012-Sep-06 17:14 UTC
[CentOS] centos email server suddenly much slower. What to do?
Greetings,
I run my own email server for some domains I administer, on a centos
vps server with a very small number of users.
The only services are smtp, imap/pop, webmail
Everything was running without problems until this morning. I left
home for 1/2 hours, and when I came back everything had become about
100x slower (seriously!). The services were/are still all up and
running, but practically unusable (even running mutt in my ssh session
is almost frozen).
I have not changed/updated anything in the last 1/2 weeks. I have
already done a few checks with the VPS provider, and it looks like:
network & hardware are OK
there seem to be no strange processes running. I didn't manage to save
the output of "top", but it didn't show anything that (AFAICT, of
course) should not be there on an email server
however, there is something that is using "much more memory than
normal" (see the comment below from the hosting sysadmin after he
checked user_beancounters). Initially we thought it was apache, but
even switching it off didn't change anything. What now?
Any help to understand what the heck happened, and find out what
exactly _started_ to cause this problem is very welcome!
TIA,
Marco
root at vps [/etc/sysconfig]# cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld
barrier limit failcnt
712: kmemsize 17208298 162267136
2147483646 2147483646 0
lockedpages 0 8
999999 999999 0
privvmpages 64694 262143
262144 262144 40
shmpages 14 2366
131072 131072 0
dummy 0 0
0 0 0
numproc 63 237
999999 999999 0
physpages 79977 262204
0 262144 0
vmguarpages 0 0
131072 2147483647 0
oomguarpages 30261 44087
131072 2147483647 0
numtcpsock 31 243
7999992 7999992 0
numflock 9 20
999999 999999 0
numpty 1 1
500000 500000 0
numsiginfo 0 27
999999 999999 0
tcpsndbuf 545000 7915960
214748160 396774400 0
tcprcvbuf 507904 3981312
214748160 396774400 0
othersockbuf 21832 1229736
214748160 396774400 0
dgramrcvbuf 0 118400
214748160 396774400 0
numothersock 56 356
7999992 7999992 0
dcachesize 10775271 154640329
2147483646 2147483646 0
numfile 772 1155
23999976 23999976 0
dummy 0 0
0 0 0
dummy 0 0
0 0 0
dummy 0 0
0 0 0
numiptent 57 57
999999 999999 0
As you can see, there are some fails for the privvmpages. This means
your VPS tried to use more RAM than what is available (e.g. more than
1GB RAM). If you are only running some basic mail services on your
VPS, that's definitely not normal and you should investigate that
accordingly. We have fully checked everything for hardware and network
problems and everything is working flawlessly. In combination with the
RAM shortage errors, it is safe to conclude that there's something
within your VPS itself that's malfunctioning.
Les Mikesell
2012-Sep-06 17:40 UTC
[CentOS] centos email server suddenly much slower. What to do?
On Thu, Sep 6, 2012 at 12:14 PM, Marco Fioretti <marco.fioretti at gmail.com> wrote:> Greetings, > I run my own email server for some domains I administer, on a centos > vps server with a very small number of users. > > The only services are smtp, imap/pop, webmail > > Everything was running without problems until this morning. I left > home for 1/2 hours, and when I came back everything had become about > 100x slower (seriously!). The services were/are still all up and > running, but practically unusable (even running mutt in my ssh session > is almost frozen). > > I have not changed/updated anything in the last 1/2 weeks. I have > already done a few checks with the VPS provider, and it looks like: > > network & hardware are OK > > there seem to be no strange processes running. I didn't manage to save > the output of "top", but it didn't show anything that (AFAICT, of > course) should not be there on an email server > > however, there is something that is using "much more memory than > normal" (see the comment below from the hosting sysadmin after he > checked user_beancounters). Initially we thought it was apache, but > even switching it off didn't change anything. What now? > > Any help to understand what the heck happened, and find out what > exactly _started_ to cause this problem is very welcome!One thing to check is that the DNS servers in /etc/resolv.conf are answering quickly (dig some_name.domain @server_ip). Mail services use DNS extensively and if the first server fails there is a timeout before trying the 2nd choice. Things will still work but slower and you may end up with enough processes running to run out of RAM and start swapping. Also check your outbound mail queue in case some spam attempt has succeeded in generating bounces. -- Les Mikesell lesmikesell at gmail.com
Marco Fioretti
2012-Sep-06 18:46 UTC
[CentOS] Fwd: centos email server suddenly much slower. What to do?
sorry, didn't realize I had not sent it to the list too. I have ran the command again several times, and now it is much slower, ~30/40 msec ---------- Forwarded message ---------- From: Marco Fioretti <marco.fioretti at gmail.com> Date: 2012/9/6 Subject: Re: [CentOS] centos email server suddenly much slower. What to do? To: Steve Clark <sclark at netwolves.com> 2012/9/6 Steve Clark <sclark at netwolves.com>:> On 09/06/2012 01:58 PM, Marco Fioretti wrote: > > 2011 msec > > > Pretty slow my dig to the same server ran in 113 msec >yes, >2 seconds seemed high to me too. But what does this mean? What can the reason be? Thx Marco
Laurent
2012-Sep-06 21:42 UTC
[CentOS] centos email server suddenly much slower. What to do?
Le 2012-09-06 19:14, Marco Fioretti a ?crit?:> Greetings, > I run my own email server for some domains I administer, on a centos > vps server with a very small number of users. > > The only services are smtp, imap/pop, webmail >> however, there is something that is using "much more memory than > normal" (see the comment below from the hosting sysadmin after he > checked user_beancounters). Initially we thought it was apache, but > even switching it off didn't change anything. What now?My 2 cents. You probably checked a lot of things. - a filesystem (almost) full ? - did you check the logs ? any errors ? - a user sending/receiving a large e-mail ? what is the maximum size of an e-mail in your MTA settings ? - more mail-services-related processes ? - did you try to stop mail services to see if the server usability is back ? - do you run antispam and/or antivirus on incoming/outgoing e-mails ? - to help with DNS, you can probably enable nscd or setup dnsmasq, so it would reduce DNS queries sent to DNS servers. -- Laurent.
M. Fioretti
2012-Sep-07 09:09 UTC
[CentOS] update on: Centos email server suddenly much slower. What to do?
On Thu, September 6, 2012 7:14 pm, Marco Fioretti wrote:> Greetings, > I run my own email server for some domains I administer, on a centos > vps server with a very small number of users. > > The only services are smtp, imap/pop, webmail > > Everything was running without problems until this morning. I left > home for 1/2 hours, and when I came back everything had become about > 100x slower (seriously!).this morning everything is back to normal (for now at least, strongly crossing my fingers!), as misteriously as it had frozen yesterday. I am able to write from my main email address just because of that. If anything, I am even more puzzled than I was yesterday. I do have the feeling, after the exchange we had yesterday, that the DNS servers my VPS provider told me to use had some problem now fixed, but nothing more. Of course, the idea that I don't know for sure what happened and it may happen again doesn't make me happy, but I honestly wouldn't know what/where to investigate at this point. Further comments are welcome. Thanks to all who helped, Marco
Lamar Owen
2012-Sep-08 11:35 UTC
[CentOS] update on: Centos email server suddenly much slower. What to do?
On Friday, September 07, 2012 05:09:09 AM M. Fioretti wrote:> I do have the feeling, after the exchange we had yesterday, that the DNS > servers my VPS provider told me to use had some problem now fixed, but > nothing more. Of course, the idea that I don't know for sure what happened > and it may happen again doesn't make me happy, but I honestly wouldn't > know what/where to investigate at this point. Further comments are > welcome.Did you say 'VPS'? It may be that other VPS instances on the host could impact your VPS performance, and there may be nothing you can do about it.