Hi Guys, Can I use MPXIO/STMS with ZFS to do multipathing amount pools/devices ? Any issues, any specific version of STMS to avoid/use ? Thanks Arif
On Wed, 16 Jul 2008, Arif Khan wrote:> Hi Guys, > Can I use MPXIO/STMS with ZFS to do multipathing amount pools/devices ? > > Any issues, any specific version of STMS to avoid/use ?By STMS I assume that you are talking about MPXIO. Solaris 10 comes with a quite usable MPXIO and it does work great with ZFS. MPXIO manages the devices so that ZFS only sees one path to each LUN. Old versions did not seem to support SAS but recent releases do. The ability to true load share or do active/standby depends on the model that the storage array uses. MPXIO is quite ugly and rough around the edges (at least compared with ZFS) but it works. Bob
Arif Khan wrote:> Hi Guys, > Can I use MPXIO/STMS with ZFS to do multipathing amount pools/devices ?Yes. It "just works" (tm).> Any issues, any specific version of STMS to avoid/use ?One issue which I''ve come across recently is that stmsboot is not behaving itself properly when it does its reboot. I''m redesigning and reimplementing stmsboot to cope with this and a few other things as well. My recommendation: (1) only have zpools in your host (2) run stmsboot -e. DON''T reboot when it asks you to (3) reboot, and make sure you add the -B \$ZFS-BOOTFS arg to the command line. (Can''t recall the sparc version, sorry) (4) when you system comes back up to single-user and tells you that it can''t mount root, run svcadm disable mpxio-upgrade (5) either hit ^D (ctrl D) or reboot James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Bob Friesenhahn wrote:> On Wed, 16 Jul 2008, Arif Khan wrote: > >> Hi Guys, >> Can I use MPXIO/STMS with ZFS to do multipathing amount pools/devices ? >> >> Any issues, any specific version of STMS to avoid/use ? > > By STMS I assume that you are talking about MPXIO. Solaris 10 comes > with a quite usable MPXIO and it does work great with ZFS. MPXIO > manages the devices so that ZFS only sees one path to each LUN. Old > versions did not seem to support SAS but recent releases do.You''re right, we added MPxIO support for SAS to S10 via patch 125081/125082, rev -14 or later. ...> MPXIO is quite ugly and rough around the edges (at least compared with > ZFS) but it works.Just curious - what do you see as the ugliness in MPxIO? I don''t have an agenda to push, I''d just like to get feedback from you on what you see as opportunities for improvement. thankyou, James -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
On Thu, 17 Jul 2008, James C. McPherson wrote:> ... >> MPXIO is quite ugly and rough around the edges (at least compared with >> ZFS) but it works. > > Just curious - what do you see as the ugliness in MPxIO? I don''t > have an agenda to push, I''d just like to get feedback from you > on what you see as opportunities for improvement.The most obvious thing is the ungainly long device names and the odd requirement to update /etc/vfstab. The other thing I noticed is that while ''mpathadm'' works as documented, it lacks in user friendlyness. It almost always requires executing a command to show what is available, and then doing a specific query based on the items listed. A few ''composite'' type commands which list the devices along with certain commonly requested information would be useful. Currently we get a list of logical units and then we do ''show lu'' to dump massive amounts of detail. When there are large amounts of logical units (as I have) this approach becomes overwelming if all I want to learn is the current path status for the devices in a ZFS pool. When I first used ''stmsboot -e'' (as the StorageTek 2540 docs recommend), it caused my system not to boot since it uses SAS disks and there were apparently problems with local SAS disks at that time (and it sounds like there still is). I learned the hard way to only request multipath for what actually needs it. Here is a small work-around script I coded up recently to list path status: #!/bin/sh # Test path access to multipathed devices devs=`mpathadm list lu | grep /dev/rdsk/` for dev in $devs do echo "=== $dev ===" mpathadm show lu $dev | grep ''Access State'' done It should provide a small example of useful composite functionality (which can obviously be far better than my script!). Bob
Hi Bob, thanks for the quick response. Comments inline below Bob Friesenhahn wrote:> On Thu, 17 Jul 2008, James C. McPherson wrote: >> ... >>> MPXIO is quite ugly and rough around the edges (at least compared >>> with ZFS) but it works. >> >> Just curious - what do you see as the ugliness in MPxIO? I don''t >> have an agenda to push, I''d just like to get feedback from you >> on what you see as opportunities for improvement. > > The most obvious thing is the ungainly long device names and the odd > requirement to update /etc/vfstab.I''m fairly sure that the long device names aspect won''t change. I don''t understand what you mean by "Odd requirement to update /etc/vfstab" - when we turn on mpxio the device paths change, so any fs that''s not ZFS will require repointing, as it were. One of the issues I''ve come up against with stmsboot and ZFSroot is that stmsboot has no clue whatsoever on how to deal with ZFS, so that''s something I''m building in to the changes I''m making. > The other thing I noticed is that> while ''mpathadm'' works as documented, it lacks in user friendlyness.[snip] I totally agree. Some time ago I logged a bug against mpathadm''s command line behaviour but got no satisfactory response from the group which maintains it.> When I first used ''stmsboot -e'' (as the StorageTek 2540 docs recommend), > it caused my system not to boot since it uses SAS disks and there were > apparently problems with local SAS disks at that time (and it sounds > like there still is). I learned the hard way to only request multipath > for what actually needs it.I''m the bloke on the hook for the S10 MPxIO/mpt backport since I''m the one who integrated the changes into the relevant gate. Could you give me more detail please on what problems you saw with the ST2540? While you''re at it, if you''ve got any suggestions that you''d like me to consider with my reimplementation of stsmboot then please let me know. Cheers, James -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
On Thu, 17 Jul 2008, James C. McPherson wrote:> I''m fairly sure that the long device names aspect won''t change. > > I don''t understand what you mean by "Odd requirement to update /etc/vfstab" > - when we turn on mpxio the device paths change, so any fs that''s not > ZFS will require repointing, as it were. One of the issues I''ve comeIt was a design choice to change the device paths. There are other approaches which would not have necessitated device path changes.> I''m the bloke on the hook for the S10 MPxIO/mpt backport since I''m the > one who integrated the changes into the relevant gate. Could you give > me more detail please on what problems you saw with the ST2540?There were no problems with the ST2540. The issues were with the SAS drives in the local backplane (Sun Ultra 40 M2). Bob
Bob Friesenhahn wrote:> On Thu, 17 Jul 2008, James C. McPherson wrote: >> I''m fairly sure that the long device names aspect won''t change. >> >> I don''t understand what you mean by "Odd requirement to update /etc/vfstab" >> - when we turn on mpxio the device paths change, so any fs that''s not >> ZFS will require repointing, as it were. One of the issues I''ve come > > It was a design choice to change the device paths. There are other > approaches which would not have necessitated device path changes.What would you have done instead?>> I''m the bloke on the hook for the S10 MPxIO/mpt backport since I''m the >> one who integrated the changes into the relevant gate. Could you give >> me more detail please on what problems you saw with the ST2540? > > There were no problems with the ST2540. The issues were with the SAS > drives in the local backplane (Sun Ultra 40 M2).Bob, I actually do want to know the specifics. As I mentioned, I''m the person who delivered the backport of those features. That means that I need to followup on issues such as the one you are alluding to. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
On Thu, 17 Jul 2008, James C. McPherson wrote:>>> >>> I don''t understand what you mean by "Odd requirement to update >>> /etc/vfstab" >>> - when we turn on mpxio the device paths change, so any fs that''s not >>> ZFS will require repointing, as it were. One of the issues I''ve come >> >> It was a design choice to change the device paths. There are other >> approaches which would not have necessitated device path changes. > > What would you have done instead?I was not involved in the design analysis so I can not say what I would have done based on all factors considered. However, there was the alternative of renaming the proxied devices. I am not saying that Sun made a wrong decision with its approach. Renaming many devices is troublesome. Covering up the low-level devices would not have been correct either.>> There were no problems with the ST2540. The issues were with the SAS >> drives in the local backplane (Sun Ultra 40 M2). > > Bob, I actually do want to know the specifics. As I mentioned, > I''m the person who delivered the backport of those features. > That means that I need to followup on issues such as the one > you are alluding to.This happened to me back in February so I don''t have specifics any more. It is worthwhile testing on a populated Sun Ultra 40 M2 or other systems using the same SAS controller. It seems likely that the problem I encountered is fixed by now since there are five or six Sun servers which use the same SAS controller. Bob