libvirt users - Jun 2012 - Storage Pools & nodedev-create specific scsi_host#?

Hello everyone,

Current host build is RHEL 6.2, soon to be upgrading.

I'm in the process of mapping out a KVM/RHEV topology.  I have questions
about the landscape of storage pools and the instantiation of specific scsi_host
IDs using virsh nodedev-create and some magic XML definitions.  I'm grouping
these questions together because the answer to one may impact the other.

High-level goals:

- fence off I/O by providing NPIV paths to VMs individually
- provide each VM with a pair of paths to the same storage for redundancy

Problem 1 - Storage Pools:

I've successfully created a storage pool by starting with an initial SCSI
host based off one of my virtual HBAs.  I see my LUNs.  I performed a virsh
pool-dumpxml and captured the configuration of the storage pool using just one
SCSI host.  I then modified the XML file to include a 2nd SCSI host, stopped
& deleted the original storage pool, then created & started a new
storage pool.  In the GUI, I am able to see the same LUNs, listed twice (once
for each path).

Here's my XML file:

<pool type='scsi'>
  <name>VM1POOL</name>
  <uuid>f0465f3f-3c9c-766c-3c96-678a99ec81cb</uuid>
  <capacity>144813588480</capacity>
  <allocation>144813588480</allocation>
  <available>0</available>
  <source>
    <adapter name='host5'/>
  </source>
  <source>
    <adapter name='host6'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0700</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

Question 1 - Is this the correct approach for presenting a VM with SAN storage
in a redundant fashion?  Or should I instead use the mpath approach and find
some way with the storage pool XML file to restrict which mpath devices are
utilized (thereby restricting which LUNs the storage pool sees?).

It seems my current approach provides each LUN twice, however a LUN has two
different volume identifiers based on its path:

UNIT:0:0:13
UNIT:0:1:13

Problem 2 - Specifying scsi_host# in a device XML file

As I mentioned, I've successfully created a virtual HBA using virsh
nodedev-create and an XML config.  Here's a sample config:

[root at host ~]# cat vm1vhba1.xml
<device>
  <parent>scsi_host3</parent>
  <capability type='scsi_host'>
    <capability type='fc_host'>
      <wwnn>a97a3b9220010002</wwnn>
      <wwpn>a97a3b9220010001</wwpn>
    </capability>
    <capability type='vport_ops' />
  </capability>
</device>

Pretty straightforward.  The parent is physical SCSI host 3.  The resulting SCSI
host is 5.  This is one of the SCSI hosts used in the storage pool above.

So, I wanted to test what would happen to my VM if I ran virsh nodedev-destroy
scsi_host5.   Essentially, testing the redundancy of the paths to the VM.  The
device was successfully deleted, however a few other things happened, both
expected and unexpected.

- Since I only associated one of the two LUN units from my storage pool with the
VM, I expected this to cause the VM to fail.  It did fail, however in a headless
fashion.  Unexpected.  The VM is still up and running in memory.  Any effort to
write out to the disks results in failures like "read-only
filesystem".  Expected.
- The Storage Pool entered into an inactive state.  Unexpected.  Efforts to
restart the storage pool using virsh pool-start failed because one of the
scsi_host devices was not present.  Expected.

I tried to recreate the device using virsh nodedev-create vm1vhba1.xml, and the
device was created successfully but as scsi_host9 (the next available scsi_host
beyond the original scsi_host5).

So, unless I can recreate this scsi_host5, I won't be able to restart my
storage pool and restore I/O to my VM.

Question 2:  Is there a definition I can use in the XML config to specify the
resulting scsi_host #?

I've been told that RHEL 6.3 is supposed to provide some methodology of
auto-instantiating my virtual HBAs upon the reboot of a host so that the VMs on
the will see their storage and can start automatically on boot.  I have a
feeling that the answer to my question may be brushing up against this
functionality.

I am also assuming that if it's possible to specify the scsi_host# in my XML
config, that I'll be able to restart my storage pool successfully and that
will enable my VM to reestablish the connection to their storage.

Thanks in advance,
MD

On 2012?06?29? 08:16, mdecheser at comcast.net wrote:
> Hello everyone,
>
> Current host build is RHEL 6.2, soon to be upgrading.
>
> I'm in the process of mapping out a KVM/RHEV topology.  I have
questions about the landscape of storage pools and the instantiation of specific
scsi_host IDs using virsh nodedev-create and some magic XML definitions. 
I'm grouping these questions together because the answer to one may impact
the other.
>
> High-level goals:
>
> - fence off I/O by providing NPIV paths to VMs individually
> - provide each VM with a pair of paths to the same storage for redundancy
>
> Problem 1 - Storage Pools:
>
> I've successfully created a storage pool by starting with an initial
SCSI host based off one of my virtual HBAs.  I see my LUNs.  I performed a virsh
pool-dumpxml and captured the configuration of the storage pool using just one
SCSI host.  I then modified the XML file to include a 2nd SCSI host,
stopped&  deleted the original storage pool, then created&  started a
new storage pool.  In the GUI, I am able to see the same LUNs, listed twice
(once for each path).
It sounds to me a bug. But I may be wrong as not farmiliar with
the scsi storage backend yet.
> Here's my XML file:
>
> <pool type='scsi'>
>    <name>VM1POOL</name>
>    <uuid>f0465f3f-3c9c-766c-3c96-678a99ec81cb</uuid>
>    <capacity>144813588480</capacity>
>    <allocation>144813588480</allocation>
>    <available>0</available>
>    <source>
>      <adapter name='host5'/>
>    </source>
>    <source>
>      <adapter name='host6'/>
>    </source>
>    <target>
>      <path>/dev/disk/by-path</path>
>      <permissions>
>        <mode>0700</mode>
>        <owner>-1</owner>
>        <group>-1</group>
>      </permissions>
>    </target>
> </pool>
>
> Question 1 - Is this the correct approach for presenting a VM with SAN
storage in a redundant fashion?  Or should I instead use the mpath approach and
find some way with the storage pool XML file to restrict which mpath devices are
utilized (thereby restricting which LUNs the storage pool sees?).
>
> It seems my current approach provides each LUN twice, however a LUN has two
different volume identifiers based on its path:
>
> UNIT:0:0:13
> UNIT:0:1:13
>
> Problem 2 - Specifying scsi_host# in a device XML file
>
> As I mentioned, I've successfully created a virtual HBA using virsh
nodedev-create and an XML config.  Here's a sample config:
>
> [root at host ~]# cat vm1vhba1.xml
> <device>
>    <parent>scsi_host3</parent>
>    <capability type='scsi_host'>
>      <capability type='fc_host'>
>        <wwnn>a97a3b9220010002</wwnn>
>        <wwpn>a97a3b9220010001</wwpn>
>      </capability>
>      <capability type='vport_ops' />
>    </capability>
> </device>
>
> Pretty straightforward.  The parent is physical SCSI host 3.  The resulting
SCSI host is 5.  This is one of the SCSI hosts used in the storage pool above.
>
> So, I wanted to test what would happen to my VM if I ran virsh
nodedev-destroy scsi_host5.   Essentially, testing the redundancy of the paths
to the VM.  The device was successfully deleted, however a few other things
happened, both expected and unexpected.
>
> - Since I only associated one of the two LUN units from my storage
pool with the VM,I expected this to cause the VM to fail.  It did fail,
however in a headless fashion. Unexpected.  The VM is still up and running
in memory.  Any effort to write out to the disks results in failures like
"read-only filesystem".  Expected.

The default write error policy is "report", so that's expected you
see
the error, while the VM is still running, even the device is destroyed.
And generally filesystem will be readonly if the underlying physical
device is detected by kernel as broken or something else.

You can set the error_policy of the vm to "stop" or
"ignore". With
error_policy="stop", you will see the VM will be stopped in this case.

See http://libvirt.org/formatdomain.html#elementsDisks for more details.
> - The Storage Pool entered into an inactive state.  Unexpected.
Have you restarted or reload libvirtd? I have no other idea of what
trigers the pool refreshing otherwise. We don't integrate with things
like inotify with storage driver yet.

Efforts to restart the storage pool using virsh pool-start failed 
because one of the scsi_host devices was not present.  Expected.

Yes, that's expected.
>
> I tried to recreate the device using virsh nodedev-create vm1vhba1.xml, and
the device was created successfully but as scsi_host9 (the next available
scsi_host beyond the original scsi_host5).
>
> So, unless I can recreate this scsi_host5, I won't be able to restart
my storage pool and restore I/O to my VM.
That's auto-numbered by scsi driver I think, not sure if there is way
to give a fix name for the new created vhba.
>
> Question 2:  Is there a definition I can use in the XML config to specify
the resulting scsi_host #?
>
> I've been told that RHEL 6.3 is supposed to provide some methodology of
auto-instantiating my virtual HBAs upon the reboot of a host so that the VMs on
the will see their storage and can start automatically on boot.  I have a
feeling that the answer to my question may be brushing up against this
functionality.
>
> I am also assuming that if it's possible to specify the scsi_host# in
my XML config, that I'll be able to restart my storage pool successfully and
that will enable my VM to reestablish the connection to their storage.
>
As far as I known, no support for this yet.

Regards,
Osier