zfs discuss - Feb 2010 - disk devices missing but zfs uses them ?

Hello I have a strange issue, 

I''m having a setup with 24 disk enclosure connected with LSI3801-R. I
created two pools. Pool have 24 healthy disks.

I disabled LUN persistency on the LSI adapter. 

When I cold boot the server (power off by pulling all power cables), a warning
is shown on the console during boot:

....
WARNING: /pci/path/ ... (mpt0): 
	wwn for traget has changed
WARNING: /pci/path/ ... (mpt0): 
	wwn for traget has changed
...

It seems that the disk take longer then the enclosure service device SES to
boot.

SAS1068E''s links are 3.0 G, 3.0 G, 3.0 G, 3.0 G, down, down, down, down

 B___T     SASAddress     PhyNum  Handle  Parent  Type
        500605b0015a5130           0001           SAS Initiator
        500605b0015a5131           0002           SAS Initiator
        500605b0015a5132           0003           SAS Initiator
        500605b0015a5133           0004           SAS Initiator
        500605b0015a5134           0005           SAS Initiator
        500605b0015a5135           0006           SAS Initiator
        500605b0015a5136           0007           SAS Initiator
        500605b0015a5137           0008           SAS Initiator
        50030480003ac2ff     0     0009    0001   Edge Expander
 0   1  50030480003ac2c4     4     000a    0009   SATA Target
 0   2  50030480003ac2c5     5     000b    0009   SATA Target
 0   3  50030480003ac2c6     6     000c    0009   SATA Target
 0   4  50030480003ac2c7     7     000d    0009   SATA Target
 0   5  50030480003ac2c8     8     000e    0009   SATA Target
 0   8  50030480003ac2c9     9     000f    0009   SATA Target
 0   6  50030480003ac2ca    10     0010    0009   SATA Target
 0   7  50030480003ac2cb    11     0011    0009   SATA Target
 0  12  50030480003ac2cc    12     0012    0009   SATA Target
 0  10  50030480003ac2cd    13     0013    0009   SATA Target
 0  11  50030480003ac2ce    14     0014    0009   SATA Target
 0   9  50030480003ac2cf    15     0015    0009   SATA Target
 0  13  50030480003ac2d0    16     0016    0009   SATA Target
 0  14  50030480003ac2d1    17     0017    0009   SATA Target
 0  15  50030480003ac2d2    18     0018    0009   SATA Target
 0  16  50030480003ac2d3    19     0019    0009   SATA Target
 0  19  50030480003ac2d4    20     001a    0009   SATA Target
 0  20  50030480003ac2d5    21     001b    0009   SATA Target
 0  17  50030480003ac2d6    22     001c    0009   SATA Target
 0  18  50030480003ac2d7    23     001d    0009   SATA Target
 0  21  50030480003ac2d8    24     001e    0009   SATA Target
 0  22  50030480003ac2d9    25     001f    0009   SATA Target
 0  23  50030480003ac2da    26     0020    0009   SATA Target
 0  24  50030480003ac2db    27     0021    0009   SATA Target
 0   0  50030480003ac2fd    36     0022    0009   SAS Initiator and Target

Then I login to the system, I see 22 disks only with format, altought all pools
are healthy and contain 24 disks.
> zpool status | grep c0 | awk ''{print $1}'' | sed -e
"s/[[:space:]]//g" -e "s/c[0-9]*t\([0-9]*\)d[0-9]*/\1 \0/g"
| sort -n
24 (this is OK, pool has 24 disks)

I did a scrub and still the pool is healthy, all disks there.
> format
Shows 22 disks two disks are missing.  (this is STRANGE)

ls -al /dev/dsk/c0t8d0 says "no such file or directory"
ls -al /dev/dsk/c0t7d0 says "no such file or directory"

Then I use zdb to find the phy path. It is: 

/devices/pci at 0,0/pci8086,3410 at 9/pci1000,3140 at 0/sd at 7,0:a
/devices/pci at 0,0/pci8086,3410 at 9/pci1000,3140 at 0/sd at 8,0:a

When I now issue a 
> ls -al /devices/pci at 0,0/pci8086,3410 at 9/pci1000,3140 at 0/sd at 7,0:a
no such file or directory 
> ls -al /devices/pci at 0,0/pci8086,3410 at 9/pci1000,3140 at 0/
.. lots of devices ..
> ls -al /devices/pci at 0,0/pci8086,3410 at 9/pci1000,3140 at 0/sd at 7,0:a
brw-r----- 1 root sys 230, 640 Feb  7 13:28 /devices/pci at 0,0/pci8086,3410 at
9/pci1000,3140 at 0/sd at 7,0:a

Strange ?? - a "ls" created device links. Is this normal ? 

When I run "format" now, I can see all 24 disks. 

My assumtion is that only the device files are missing, however the device (in
kernel) is there and used.

How can ZFS use a device that does not exist on the system ?
-- 
This message posted from opensolaris.org