CentOS - Feb 2016 - Strange performance issue on CentOS 6.7 server

I?m running CentOS 6.7 on my build servers, and on one of the servers the builds
are taking almost an order of magnitude longer than usual.  There are no runaway
processes and there is plenty of free memory.  So I suspected that file I/O
might be slow, and sure enough, that appears to be the case.  I ran a simple dd
test and compared the results to a ?normal? build server (412 MB/s vs. 31.7
MB/s).  Both are on similarly configured VGs on identical hardware.  What could
cause this performance degradation and are there any other tests I can run
before I reboot the server to see if things improve?  There is nothing in
/var/log messages - are there other logs I should check?

Alfred

On Feb 3, 2016, at 1:30 PM, Alfred von Campe <alfred at von-campe.com>
wrote:
> 
> I suspected that file I/O might be slow, and sure enough, that appears to
be the case?.What could cause this
A dying hard disk can do it.  HDDs try to silently paper over I/O errors, but
what they can?t hide is the time it takes to do this.  If your HDD is constantly
correcting errors at the oxide layer, it will be reeeeeallly sllllow.

You can try running SMART tests on it, though that?s not guaranteed to show the
problem.

Got tested backups? :)

On Feb 3, 2016, at 16:13, Warren Young wrote:
> A dying hard disk can do it.  HDDs try to silently paper over I/O errors,
but what they can?t hide is the time it takes to do this.  If your HDD is
constantly correcting errors at the oxide layer, it will be reeeeeallly sllllow.
> 
> You can try running SMART tests on it, though that?s not guaranteed to show
the problem.
Well, it?s not ?a? disk: it?s a HW RAID of about dozen (server grade) drives,
with a VG/LV on top of that.  Are there any log files I can check that test the
underlying VG/LV health status?

Alfred