CentOS - Sep 2009 - SUMMARY : multipath using defaults rather than multipath.conf contents for some devices (?) - why ?

The reason for the behaviour observed below turned out to be that the 
device  entry in /etc/multipath.conf was inadvertently appended *after* the
devices
section , rather  than inside it - so that we  had

#devices {
#    device {
#        blah blah
#    }                 (file has a bunch of defaults commented out)
#    etc
#}
#              
#    
device {
  our settings
}

*rather than*

devices {
       device {
  		our settings
	}
}


Also - looking more closely at our multipath.conf.defaults, there is an entry
for
a product pattern HSV2.*

That would explain why the multipath settings for the HSV200 looked different to
the HSV400 - the HSV200 was picking up a different set of defaults.
However still not exactly what we had specified.

(sort of like CSS behaviour....."cascading multipath.conf defaults" !
:-) )

You also possibly need to pay attention to the basic whitespace formatting of
this
file - for example I had noticed the above early on and rebooted
with it "fixed" , only to find that this meant the system came up 
unable to even recognise the ext3 filesystem, complaining about bad superblock 
etc etc.

After recovering from that ( see below)  I went back ( a week later) and a) made
sure that
the  whitespace - tabs etc  - looked like the  other commented out defaults and
b) added
in a second device entry (from the defaults section - not a device we even have)
, just
in case there was some bug relating to having only one device entry in the
"devices"
section.

Whether because of those or some other change, the system now comes up fine, and
with
multipath -ll now reporting the correct settings for the HSV400. So issue
resolved
finally. Touch wood.

Here is a useful tip (not news to gurus but was to me ) : when things turn to
custard on reboot and
everything including the root filesystem is mounted readonly - so you can't
even restore multipath.conf
to the  backup you made before rebooting - you can 


mount -n -o remount /

to remount / as writeable - then you can 

cd /etc
cp multipath.conf.bu1 multipath.conf

and breathe a big sigh of relief !

If there are any multipath developers reading this - it could be handy if
multipath could log some
diagnostic info about how it parses the conf file , and exactly what entry it
ends up using , which
would then appear in /var/log/messages and /var/log/dmesg, since it seems
relatively easy to end up
matching a device spec you didn't expect, and the effects on performance are
subtle so
easy to overlook this.


Cheers

AMcC


------------------------ original post ------------------------

hi all

We have a rh linux server connected to two HP SAN controllers, one an HSV200 (on
the way out),
the other an HSV400 (on the way in). (Via a Qlogic HBA).

/etc/multipath.conf contains this : 

device
{
        vendor          "(COMPAQ|HP)"
        product         "HSV1[01]1|HSV2[01]0|HSV300|HSV4[05]0"
        getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout    "/sbin/mpath_prio_alua /dev/%n"
        hardware_handler "0"
        path_selector   "round-robin 0"
        path_grouping_policy    group_by_prio
        failback        immediate
        rr_weight       uniform
        no_path_retry   18
        rr_min_io       100
        path_checker    tur
}

- but our actual multipathing as shown by multipath -ll , and multipath -ll -v 3
looks as though for the
HSV400 it is using the defaults rather  than these settings. The defaults are 

#defaults {
#       udev_dir                /dev
#       polling_interval        10
#       selector                "round-robin 0"
#       path_grouping_policy    multibus
#       getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
#       prio_callout            /bin/true
#       path_checker            readsector0
#       rr_min_io               100
#       rr_weight               priorities
#       failback                immediate
#       no_path_retry           fail
#       user_friendly_name      yes


and multipath -ll reports :

.
.
[snip other HSV400 paths - all similar]
mpath12 (3600508b40007518f0000900000520000) dm-1 HP,HSV400
[size=150G][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
 \_ 0:0:5:9  sdab 65:176 [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 0:0:3:9  sdn  8:208  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 0:0:4:9  sdu  65:64  [active][ready]
mpath11 (3600508b40007518f0000700000370000) dm-6 HP,HSV200
[size=200G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=50][active]
 \_ 0:0:1:7  sdd  8:48   [active][ready]
\_ round-robin 0 [prio=10][enabled]
 \_ 0:0:2:7  sdh  8:112  [active][ready]
.
.
[snip other HSV200 paths - all similar]



multipath -ll -v 3 includes explicit statements that defaults are being used for
the HSV400

(long output snipped...)

sdaa: path checker = readsector0 (config file default)

versus

sda: path checker = tur (controller setting)

sdx: getprio = NULL (internal default)

versus

sdd: getprio = /sbin/mpath_prio_alua %n (controller setting)



- furthermore we see in the log file messages from both readsector0 *and* tur 
rather than just tur if the correct settings were used , which also backs that
up.

My questions are basically - why is it happening , and how to fix it ? 

The vendor and product regexps definitely do match both "HSP" and both
"HSV200" and "HSV400" respectively
so it doesn't seem that fiddling with the patterns will work , and I'm
sure this config has been tested.

Its not due to this server having to deal with two controllers - we have a
second server that only mounts from
the HSV400, and  its multipath settings appear to be entirely the defaults, and
not what we have set.

(And conversely, its not due to the conf file not being read at all - since the
server with two controllers
is using the correct config for one of them , but not the other.)

thanks for any tips and I will summarise.

Cheers

AMcC

======================================================================Attention:
The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
=======================================================================