zfs discuss - Jul 2009 - The zfs performance decrease when enable the MPxIO round-robin

Hi Guys
I have a SunFire X4200M2 and the Xyratex RS1600 JBOD which I try to
run the ZFS on it.But I found a problem:
I set mpxio-disable="yes" in the /kernel/drv/fp.conf to enable the
MPxIO,
and set load-balance="round-robin" in the /kernel/drv/scsi_vhci.conf
enable
the round-robin.The ZFS performance is very low, it is about 50% performance
decrease. If I disable the MPxIO or just set load-balance="none", the
performace
is good to accept.

I am confused. I google the website and found this
:http://xyratex.mobi/pdfs/products/storage-systems/tips/TIP107_Configuring_Solaris10_x86_for_Xyratex_storage_1-0.pdf

It is the document of Xyratex, they have same result but no explain.Maybe it is 
the SunMdi bug? How can I do this ?

Thanks,any suggestion is welcome!
-- 
This message posted from opensolaris.org

On Sat, 18 Jul 2009 23:25:27 -0700 (PDT)
lf yang <no-reply at opensolaris.org> wrote:
> Hi Guys
> I have a SunFire X4200M2 and the Xyratex RS1600 JBOD which I try to
> run the ZFS on it.But I found a problem:
> I set mpxio-disable="yes" in the /kernel/drv/fp.conf to enable
the MPxIO,
> and set load-balance="round-robin" in the
/kernel/drv/scsi_vhci.conf enable
> the round-robin.The ZFS performance is very low, it is about 50%
performance
> decrease. If I disable the MPxIO or just set load-balance="none",
the performace
> is good to accept.
> 
> I am confused. I google the website and found this
:http://xyratex.mobi/pdfs/products/storage-systems/tips/TIP107_Configuring_Solaris10_x86_for_Xyratex_storage_1-0.pdf
> 
> It is the document of Xyratex, they have same result but no explain.Maybe
it is
> the SunMdi bug? How can I do this ?

To enable mpxio, you need to have

mpxio-disable="no";

in your fp.conf file. You should run /usr/sbin/stmsboot -e to make
this happen. If you *must* edit that file by hand, always run
/usr/sbin/stmsboot -u afterwards to ensure that your system''s MPxIO
config is correctly updated.


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog
Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel

>
> To enable mpxio, you need to have
>
> mpxio-disable="no";
>
> in your fp.conf file. You should run /usr/sbin/stmsboot -e to make
> this happen. If you *must* edit that file by hand, always run
> /usr/sbin/stmsboot -u afterwards to ensure that your system''s
MPxIO
> config is correctly updated.
I thought stmsboot was mildly broken on the latest releases of S10 and
possibly confused in the ZFS world. I am going by memory here of course
and this may have been fixed since I looked at it 6 months ago or so. I
also feel that editing the fb.conf file manually and entering the paths
chosen is perhaps the best way to go.

Also, since I''m up late and posting a comment anyways, I set up a V890
with mpxio and ZFS with every ZFS mirror being composed of a mpxio enabled
device. The whole process took some time but the level of redundency and
throughput was worth it.  Who knows, maybe someday mpxio will be default
from install on FCAL enabled machines.

Dennis

ps: I''m going to go search for those bugids if they ever existed

On Sun, 19 Jul 2009 02:56:47 -0400 (EDT)
Dennis Clarke <dclarke at blastwave.org> wrote:
> 
> >
> > To enable mpxio, you need to have
> >
> > mpxio-disable="no";
> >
> > in your fp.conf file. You should run /usr/sbin/stmsboot -e to make
> > this happen. If you *must* edit that file by hand, always run
> > /usr/sbin/stmsboot -u afterwards to ensure that your system''s
MPxIO
> > config is correctly updated.
> 
> I thought stmsboot was mildly broken on the latest releases of S10 and
> possibly confused in the ZFS world. I am going by memory here of course
> and this may have been fixed since I looked at it 6 months ago or so. I
> also feel that editing the fb.conf file manually and entering the paths
> chosen is perhaps the best way to go.

It''s not broken any more, because I fixed it.


http://blogs.sun.com/jmcp/entry/on_stmsboot_1m

6707555 stmsboot is lost in a ZFS root world  

patched in 140334-01 and 140335-01. Rolled into S10 Update 7.


There have been some followon fixes.

There is no reason to edit fp.conf or mpt.conf yourself, unless you
must use per-hba MPxIO. In which case, make darned sure you run
/usr/sbin/stmsboot -u afterwards.

 
> Also, since I''m up late and posting a comment anyways, I set up a
V890
> with mpxio and ZFS with every ZFS mirror being composed of a mpxio enabled
> device. The whole process took some time but the level of redundency and
> throughput was worth it.  Who knows, maybe someday mpxio will be default
> from install on FCAL enabled machines.
> ps: I''m going to go search for those bugids if they ever existed
In a pre-FCS build of S10, MPxIO was enabled by default, but there
were some problems with jumpstart as a result, so the default was
changed back. On X86 with fc, MPxIO is the default, but that is the
only case where that is true at this point in time.


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog
Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel

On Sun, Jul 19, 2009 at 08:25, lf yang<no-reply at opensolaris.org>
wrote:
> Hi Guys
> I have a SunFire X4200M2 and the Xyratex RS1600 JBOD which I try to
> run the ZFS on it.But I found a problem:
> I set mpxio-disable="yes" in the /kernel/drv/fp.conf to enable
the MPxIO,
I assume you mean mpxio-disable="no"

> and set load-balance="round-robin" in the
/kernel/drv/scsi_vhci.conf enable
> the round-robin.The ZFS performance is very low, it is about 50%
performance
> decrease. If I disable the MPxIO or just set load-balance="none",
the performace
> is good to accept.
>
> I am confused. I google the website and found this
:http://xyratex.mobi/pdfs/products/storage-systems/tips/TIP107_Configuring_Solaris10_x86_for_Xyratex_storage_1-0.pdf
>
> It is the document of Xyratex, they have same result but no explain.Maybe
it is
> the SunMdi bug? How can I do this ?
That is probably a limitation in the hardware.