We''re planning to build a ZFS-based Solaris NFS fileserver environment with the backend storage being iSCSI-based, in part because of the possibilities for failover. In exploring things in our test environment, I have noticed that ''zpool import'' takes a fairly long time; about 35 to 45 seconds per pool. A pool import time this slow obviously has implications for how fast we can import a bunch of pools during a failover situation, so I''d like to speed it up somehow (ideally in non-hacky ways). (Trying to do all of the ''zpool import''s in parallel doesn''t seem to speed the collective set of them up relative to doing them sequentially.) My test environment currently has 132 iSCSI LUNs (and 132 pools, one per LUN, because I wanted to test with extremes) on an up to date S10U4 machine. A truss of a ''zpool import'' suggests that it spends most of its time opening various disk devices and reading things from them and most of the rest of the time doing modctl() calls and ZFS ioctls(). (Also, using ''zpool import -d'' with a prepared directory that has only symlinks to the particular /dev/dsk device entries for a pool''s LUN speeds things up dramatically.) So, are there any tricks to speeding up ZFS pool import here (short of the ''zpool import -d'' stuff)? Would Sun Cluster manage this faster, or does its ZFS pool failover stuff basically reduce to ''zpool import'' too? Thanks in advance. - cks
Look at alternate cachefiles (''zpool set cachefile'',
''zpool import -c
<cachefile>'', etc). This avoids scanning all devices in the
system and
instead takes the config from the cachefile.
- Eric
On Tue, May 20, 2008 at 05:20:52PM -0400, Chris Siebenmann
wrote:> We''re planning to build a ZFS-based Solaris NFS fileserver
environment
> with the backend storage being iSCSI-based, in part because of the
> possibilities for failover. In exploring things in our test environment,
> I have noticed that ''zpool import'' takes a fairly long
time; about
> 35 to 45 seconds per pool. A pool import time this slow obviously
> has implications for how fast we can import a bunch of pools during
> a failover situation, so I''d like to speed it up somehow (ideally
in
> non-hacky ways).
>
> (Trying to do all of the ''zpool import''s in parallel
doesn''t seem
> to speed the collective set of them up relative to doing them
> sequentially.)
>
> My test environment currently has 132 iSCSI LUNs (and 132 pools, one
> per LUN, because I wanted to test with extremes) on an up to date S10U4
> machine. A truss of a ''zpool import'' suggests that it
spends most of
> its time opening various disk devices and reading things from them and
> most of the rest of the time doing modctl() calls and ZFS ioctls().
>
> (Also, using ''zpool import -d'' with a prepared directory
that has only
> symlinks to the particular /dev/dsk device entries for a pool''s
LUN
> speeds things up dramatically.)
>
> So, are there any tricks to speeding up ZFS pool import here (short of
> the ''zpool import -d'' stuff)? Would Sun Cluster manage
this faster, or
> does its ZFS pool failover stuff basically reduce to ''zpool
import'' too?
>
> Thanks in advance.
>
> - cks
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Eric Schrock, Fishworks http://blogs.sun.com/eschrock
On Tue, 20 May 2008, Chris Siebenmann wrote:> We''re planning to build a ZFS-based Solaris NFS fileserver environment > with the backend storage being iSCSI-based, in part because of the > possibilities for failover. In exploring things in our test environment, > I have noticed that ''zpool import'' takes a fairly long time; about > 35 to 45 seconds per pool. A pool import time this slow obviously > has implications for how fast we can import a bunch of pools during > a failover situation, so I''d like to speed it up somehow (ideally in > non-hacky ways).How many pools is "a bunch"? The ideal number of pools per server tends to one, so reducing the number of pools might be your best option. -- Rich Teer, SCSA, SCNA, SCSECA CEO, My Online Home Inventory URLs: http://www.rite-group.com/rich http://www.linkedin.com/in/richteer http://www.myonlinehomeinventory.com
[Eric Schrock:]
| Look at alternate cachefiles (''zpool set cachefile'',
''zpool import -c
| <cachefile>'', etc). This avoids scanning all devices in the
system
| and instead takes the config from the cachefile.
This sounds great.
Is there any information on when this change will make it to Solaris?
(In particular, is it going to be in S10 update 6, or only in a later
version?)
[Rich Teer:]
| How many pools is "a bunch"? The ideal number of pools per server
| tends to one, so reducing the number of pools might be your best
| option.
At a guess, we will probably have on the order of fifty to seventy
pools per fileserver (each of them using at least two LUNs, and we
will have multiple fileservers, all of which can see all of the LUNs).
We want to use many pools because we feel that using many pools is a
simpler way to manage selling fixed-size chunks of storage to users than
putting them all in a few pools and using quotas.
(There are other situations that call for many pools; the example I
remember from an earlier discussion on zfs-discuss is someone who
expected to have a bunch of zones and wanted to be able to move each of
them between machines.)
- cks
On Wed, May 21, 2008 at 04:59:54PM -0400, Chris Siebenmann wrote:> [Eric Schrock:] > | Look at alternate cachefiles (''zpool set cachefile'', ''zpool import -c > | <cachefile>'', etc). This avoids scanning all devices in the system > | and instead takes the config from the cachefile. > > This sounds great. > Is there any information on when this change will make it to Solaris? > (In particular, is it going to be in S10 update 6, or only in a later > version?)I believe it will be in S10u6. Someone more knowledgeable will correct me if I''m wrong ;-) - Eric -- Eric Schrock, Fishworks http://blogs.sun.com/eschrock