> I'm missing something.
>
> The purpose of NPIV (as I understand it) is to give a guest OS an HBA that
> it can scan, play with new luns, etc all without making changes to the
> physical server(s) the guest is living in currently.
Technically, the purpose of NPIV is to provide discrete paths to storage. Who
uses it and how it's used are totally up for debate. I could make a case
for using NPIVs on a physical system to isolate I/O for an application mounted
at /mount/point/A that has no I/O impact on other NPIV storage mounted at
/mount/point/B. The concept of using NPIVs with VMs is a logical extension of
that idea.
> However, I can't find a way to either have the guest's XML config
create
> the HBA or for the physical server to successfully GIVE the HBA to the
> guest. I can give disks all day long, but I can't give the HBA to the
> guest.
I've already been down this path, no pun intended. You do not present a
virtual HBA to your guest. You create virtual HBAs on the host, zone/mask your
storage to your virtual HBA and present block-level devices to your VM, using
Storage Pools & virtio devices to provide passthru. Your host doesn't
use them beyond knowing there's storage zoned/masked to the vHBAs you
created for it.
I learned this by both opening a case with Red Hat as well as pressing the topic
laterally through my organization's Sales Engineer, who contacted Red
Hat's virtualization team directly.
> It seems like libvirt and virsh can only create the vHBA on the physical
> box, which defeats the purpose of working with NPIV then... I can just
> present the same luns to the REAL WWPNs of the physical boxes that need
> access, setup multipathd to give /dev/mapper names in a consistent manner,
> and give the raw disks to the guests.
Use virsh nodedev-create /path/to/your/virtualhbacfg.xml. It will create a
virtual device listed in /sys/class/scsi_host. You zone/mask storage to the
NPIVs specified in your XML file. I then created a storage pool using
virt-manager. I can see the storage using either mpath devices or iSCSI devices
in virt-manager, but I believe that the solution is to use iSCSI. This method
allows you to specify the device (choose the virtual devices), and therefore
specify the paths. Using mpath devices is not what you want because it will
look at all scsi_hosts, find all storage down all paths, and provides no path
segregation which is what I believe you're seeking.
> I really want to get the physical KVM servers out of the 'storage
> management' game aside from the basic OSes of the guests. How can I do
> that?
>
> (or am I missing something?)
I believe it's better to manage your storage in one place than in many.
You're virtualizing kernels here. Might as well virtualize your storage
devices in the same place. Let the host do the hard work and let your VMs relax
:)
Believe me, I understand your frustration. I'm working on a massive project
to move all my Linux VMs away from VMWare (for a number of reasons). In
parallel, I'm working on RHEL on IBM Power (pSeries) and developing a
KVM/RHEV infrastructure. Of all the virtualization solutions under my
organization's roof, the one that is the best engineered to date is the IBM
solution because it's the only one that decidedly leverages NPIV, virtual
HBAs (that, yes, are passed down to the VM directly) and nuke-proof redundancy.
RHEL on POWER is as resilient as AIX on POWER with one drawback ... it's a
PPC chip which drastically limits what software I can run on it. So I have
RHEL/PPC w/NPIV and then KVM w/NPIV. I am using the AIX/POWER + RHEL/POWER
design as my model as I'm developing my KVM offering.
There are similarities and differences, but the end result is essentially the
same (overlooking certain fault tolerance aspects which put the
extremely-more-expensive IBM solution ahead of the rest).
IBM: you create VMs which run on the hardware whose sole lot in life is to
handle I/O. Those specialized systems are called VIO servers. They create,
distribute and manage NICs and vHBAs. They handle the overhead for all disk I/O
and network I/O. They enable you to dynamically add/destroy virtual devices and
grant/revoke them to the rest of the VMs on the hardware. The VIO does not know
about the storage presented to it, though it knows about the NPIVs associated
with its vHBAs as well as the vLANs associated with its network devices.
KVM/Libvirt does all this as well. But instead of managing it from within a VM,
you manage it on the physical hardware. Creating your vHBAs on the hardware
doesn't really tax the physical host too much because you're leaving all
the disk traffic to the HBAs themselves. That's what they're designed
to do.
Both flavors of VM are entirely SAN-based. IBM's RHEL solution knows its
installed on SAN storage devices and boots off of them directly. KVM presents
virtual SCSI devices which the VM thinks are local disks. From the guest level,
I'd personally rather troubleshoot and work with what appear to be local
disks. From top to bottom, it's just easier to build, manage, kickstart,
etc.
I'm in the final phases of my designs with KVM. So far, I'm very
impressed with it and I've even shown my snobby UNIX / IBM colleagues a
thing or two to give them a run for their money :)
Lastly, don't beat yourself up. All of this is relatively new stuff. Red
Hat is still perfecting the methodology behind it. RHEL 6.3 is supposed to
provide auto-instantiation of vHBAs on host boot, so any VMs set to auto-start
on boot won't fail because it's waiting around for you to manually
recreate the vHBAs.
Hope this is helpful.
MD
>
> --Jason