CentOS - Feb 2016 - Strange performance issue on CentOS 6.7 server

On Feb 3, 2016, at 16:13, Warren Young wrote:
> A dying hard disk can do it.  HDDs try to silently paper over I/O errors,
but what they can?t hide is the time it takes to do this.  If your HDD is
constantly correcting errors at the oxide layer, it will be reeeeeallly sllllow.
> 
> You can try running SMART tests on it, though that?s not guaranteed to show
the problem.
Well, it?s not ?a? disk: it?s a HW RAID of about dozen (server grade) drives,
with a VG/LV on top of that.  Are there any log files I can check that test the
underlying VG/LV health status?

Alfred

Alfred von Campe wrote:
> On Feb 3, 2016, at 16:13, Warren Young wrote:
>
>> A dying hard disk can do it.  HDDs try to silently paper over I/O
>> errors, but what they can?t hide is the time it takes to do this.  If
>> your HDD is constantly correcting errors at the oxide layer, it will be
>> reeeeeallly sllllow.
>>
>> You can try running SMART tests on it, though that?s not guaranteed to
>> show the problem.
>
> Well, it?s not ?a? disk: it?s a HW RAID of about dozen (server grade)
> drives, with a VG/LV on top of that.  Are there any log files I can check
> that test the underlying VG/LV health status?
>
You don't mention what kind of h/w RAID. LSI-based controller, and
HP-based ones, both have utilities to check out the drives (MegaRAID and
hpacli, respectively); AC&NC JetStors have a web interface.

    mark

On Feb 3, 2016, at 2:26 PM, Alfred von Campe <alfred at von-campe.com>
wrote:
> 
> On Feb 3, 2016, at 16:13, Warren Young wrote:
> 
>> A dying hard disk can do it.  HDDs try to silently paper over I/O
errors, but what they can?t hide is the time it takes to do this.  If your HDD
is constantly correcting errors at the oxide layer, it will be reeeeeallly
sllllow.
>> 
>> You can try running SMART tests on it, though that?s not guaranteed to
show the problem.
> 
> Well, it?s not ?a? disk: it?s a HW RAID of about dozen (server grade)
drives
smartctl can see through several different types of RAID controller to the
underlying physical disks via its -d option.

On Feb 3, 2016, at 17:10, Warren Young wrote:
> smartctl can see through several different types of RAID controller to the
underlying physical disks via its -d option.
This is what I have:

# smartctl --all /dev/sda
smartctl 5.43 2012-06-30 r3573 [i686-linux-2.6.32-573.12.1.el6.i686] (local
build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               IBM     
Product:              ServeRAID M5110e
Revision:             3.19
User Capacity:        1,494,996,746,240 bytes [1.49 TB]
Logical block size:   512 bytes
Logical Unit id:      0x60050760408e81b018be601809efd11c
Serial number:        001cd1ef091860be18b0818e40600705
Device type:          disk
Local Time is:        Wed Feb  3 17:13:34 2016 EST
Device does not support SMART

Error Counter logging not supported
Device does not support Self Test logging

I guess I am stuck since it says it doesn?t support SMART.  Or is there some way
to get some status from this ?disk? to see if it?s really the root cause of my
performance issues.  I think I would have seen something in /var/log/messages if
there was a critical issue.

Alfred