CentOS - Oct 2008 - Possibly OT: intermittently long response times when connecting to host

I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and
drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it -
2 hosts. The problem is that when I pull up a web page after not
having accessed it for ~15+ minutes it takes between 10-12s to load
fully. The 2nd page, 3rd page - and sometimes more than that - take
between 8-12 second. After that - decent performance, though not great
(0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes
back to slow. While it's been most apparent with httpd, I've noticed
similar slowness with the initial ssh session, etc.

I know what you're thinking. DNS problem, right? Yeah, me too. But for
the life of me I can't find a problem. This is at a colocation site
where we have other servers - all of those are just fine. We've got an
almost identical box internally - working just fine. I can do an
nslookup off of each of our different internal name servers and get
identical and immediate responses.

I've done tcpdumps (both on the client and the server). It shows the
request gets to the server immediately. I've started httpd using
strace and not seen anything glaringly wrong (though I'm not well
versed in the output I have enough C under my belt to have an idea
what's going on). Thinking it was code that the developer working on
the project I pulled all the custom drupal modules out and it's still
slow. Even created another virtual host with a stock drupal install
and it's slow (identical results). I moved the mysql server from local
to our central DB server. Load avg never gets above ~0.10 so I don't
think the box is stressed. Watching the ESX load average it looks good
too. I've tried a hell of a lot of other things and tried to include
all the relevant information so apologies in advance for anything I
might have left out.


Thanks,
Dave

Relevant info:

uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP
Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux

relevant portion of httpd.conf:
NameVirtualHost *:80

<VirtualHost *:80>
   DocumentRoot /path/to/drupal-current
   ServerName publicname.domainname.com
   ErrorLog /path/to/logserror_log
   CustomLog /path/to/logs/access_log common
</VirtualHost>

/etc/hosts:
127.0.0.1 localhost.localdomain localhost internal-name
12.34.56.78 publicname.domainname.com
12.34.56.79 db-server.domainname.com db-server

On Tue, 2008-10-28 at 16:30 -0500, David Miller wrote:
> I have a CentOS 5.2 server running Apache 2.2.3 (yum installed) and
> drupal 6.5 on ESX 3.5. We're using Name Based Virtual Hosting on it -
> 2 hosts. The problem is that when I pull up a web page after not
> having accessed it for ~15+ minutes it takes between 10-12s to load
> fully. The 2nd page, 3rd page - and sometimes more than that - take
> between 8-12 second. After that - decent performance, though not great
> (0.3s to 2s). Wait ~15 minutes, maybe more, maybe less, and it goes
> back to slow. While it's been most apparent with httpd, I've
noticed
> similar slowness with the initial ssh session, etc.
> 
> I know what you're thinking. DNS problem, right? Yeah, me too. But for
> the life of me I can't find a problem. This is at a colocation site
> where we have other servers - all of those are just fine. We've got an
> almost identical box internally - working just fine. I can do an
> nslookup off of each of our different internal name servers and get
> identical and immediate responses.
> 
> I've done tcpdumps (both on the client and the server). It shows the
> request gets to the server immediately. I've started httpd using
> strace and not seen anything glaringly wrong (though I'm not well
> versed in the output I have enough C under my belt to have an idea
> what's going on). Thinking it was code that the developer working on
> the project I pulled all the custom drupal modules out and it's still
> slow. Even created another virtual host with a stock drupal install
> and it's slow (identical results). I moved the mysql server from local
> to our central DB server. Load avg never gets above ~0.10 so I don't
> think the box is stressed. Watching the ESX load average it looks good
> too. I've tried a hell of a lot of other things and tried to include
> all the relevant information so apologies in advance for anything I
> might have left out.
> 
> 
> Thanks,
> Dave
> 
> Relevant info:
> 
> uname -a: Linux internal-name.domainname.com 2.6.18-92.1.13.el5 #1 SMP
> Wed Sep 24 19:33:52 EDT 2008 i686 i686 i386 GNU/Linux
> 
> relevant portion of httpd.conf:
> NameVirtualHost *:80
> 
> <VirtualHost *:80>
>    DocumentRoot /path/to/drupal-current
>    ServerName publicname.domainname.com
>    ErrorLog /path/to/logserror_log
>    CustomLog /path/to/logs/access_log common
> </VirtualHost>
> 
> /etc/hosts:
> 127.0.0.1 localhost.localdomain localhost internal-name
> 12.34.56.78 publicname.domainname.com
> 12.34.56.79 db-server.domainname.com db-server
----
sounds sleepy - perhaps the mysql connections drop after inactivity and
then it takes that long to connect

You might just want to set up a cron script that does a curl or wget of
a standard page every 2 minutes or so just to keep things active.

Craig