zfs discuss - Oct 2009 - cachefile for snail zpool import mystery?

Hi,

We are seeing more long delays in zpool import, say, 4~5 or even
25~30 minutes, especially when backup jobs are going on in the FC SAN 
the LUNs resides (no iSCSI LUNs yet). On the same node for the LUNs of the same
array,
some pools takes a few seconds, but minutes for some. the pattern
seems random to me so far.  It''s first noticed soon after being
upgraded to
Solaris 10 U6 (10/08, on sparc, M4000,Vx90 using some IBM and Sun arrays.) 
Appreciated, if someone can comment on this. Thanks.

We have a few VCS clusters, each has a set of service groups that 
import/export some zpools at proper events on a proper node 
(with ''-R /'' option). To fix the long delays, it seems I can
use
the ''zpool set cachefile=/x/... ...'' for each pool, deploy
all cachefiles to every cluster node of a cluster on a persisent 
location,/y/, then have the agent online script do 
''zpool import -c /y/...'', if /y/... exists. Any better fix?

1. Why would it ever take so long (20-30 minutes!) to import a pool?
It seems I/O on the FC SAN were just fine, no error messages either. 
Is it problems of other stacks or because I deleted some LUNs on the array
without taking it off device trees? 

2.  we now have the burden of maintaining these cachefiles when
we change the zpool, say add/drop a lun. any advice?
It''d be nice if zfs keeps a cache file (other than
/etc/zfs/zpool.cache)
for those ones imported under an altroot, and make it persistent,
verify/update entries at proper events. At least, I wish zfs allow
us to create the cachefiles while they are not currently imported.
so that I can just have a simple daily job to maintain the cache files 
on every node of a cluster automatically. 
 
Thanks.
Max
-- 
This message posted from opensolaris.org

Max Holm wrote:
> Hi,
>
> We are seeing more long delays in zpool import, say, 4~5 or even
> 25~30 minutes, especially when backup jobs are going on in the FC SAN 
> the LUNs resides (no iSCSI LUNs yet). On the same node for the LUNs of the
same array,
> some pools takes a few seconds, but minutes for some. the pattern
> seems random to me so far.  It''s first noticed soon after being
upgraded to
> Solaris 10 U6 (10/08, on sparc, M4000,Vx90 using some IBM and Sun arrays.) 
> Appreciated, if someone can comment on this. Thanks.
>
> We have a few VCS clusters, each has a set of service groups that 
> import/export some zpools at proper events on a proper node 
> (with ''-R /'' option). To fix the long delays, it seems I
can use
> the ''zpool set cachefile=/x/... ...'' for each pool,
deploy
> all cachefiles to every cluster node of a cluster on a persisent 
> location,/y/, then have the agent online script do 
> ''zpool import -c /y/...'', if /y/... exists. Any better
fix?
>
> 1. Why would it ever take so long (20-30 minutes!) to import a pool?
> It seems I/O on the FC SAN were just fine, no error messages either. 
> Is it problems of other stacks or because I deleted some LUNs on the array
> without taking it off device trees? 
>   
This is probably your problem. Try to do devfsadm -vC

> 2.  we now have the burden of maintaining these cachefiles when
> we change the zpool, say add/drop a lun. any advice?
> It''d be nice if zfs keeps a cache file (other than
/etc/zfs/zpool.cache)
> for those ones imported under an altroot, and make it persistent,
> verify/update entries at proper events.
IIRC when you change a pool config its cache file will be automatically 
updated on the same node.
>  At least, I wish zfs allow
> us to create the cachefiles while they are not currently imported.
> so that I can just have a simple daily job to maintain the cache files 
> on every node of a cluster automatically. 
>  
>   
What you can do is to put a script in a crontab which checks if a pool 
is currently imported on this node and if it is then copy the pools 
cache file to over nodes.

btw: IIRC Sun Cluster HAS+ agane will automatically make use of cache files

-- 
Robert Milkowski 
http://milek.blogspot.com