zfs discuss - Jan 2009 - can''t create new pool: another disk has a zpool active?

# rmformat
Looking for devices...
     1. Logical Node: /dev/rdsk/c3t0d0p0
        Physical Node: /pci at 0,0/pci108e,cb84 at 2,1/storage at 1/disk at 0,0
        Connected Device: Ext Hard  Disk
        Device Type: Removable
     2. Logical Node: /dev/rdsk/c2t0d0p0
        Physical Node: /pci at 0,0/pci108e,cb84 at 2,1/storage at 2/disk at 0,0
        Connected Device: Ext Hard  Disk
        Device Type: Removable
     3. Logical Node: /dev/rdsk/c3t0d1p0
        Physical Node: /pci at 0,0/pci108e,cb84 at 2,1/storage at 1/disk at 0,1
        Connected Device: Ext Hard  Disk
        Device Type: Removable
     4. Logical Node: /dev/rdsk/c0t0d0p0
        Physical Node: /pci at 0,0/pci-ide at 6/ide at 0/sd at 0,0
        Connected Device: TEAC     DW-224SL-R       1.0A
        Device Type: DVD Reader

# zpool status
  pool: data
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          c2t0d0p0  ONLINE       0     0     0

errors: No known data errors

  pool: data2
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
        replicas for the pool to continue functioning.
action: Attach the missing device and online it using ''zpool
online''.
   see: http://www.sun.com/msg/ZFS-8000-3C
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data2       UNAVAIL      0     0     0  insufficient replicas
          c3t0d0    UNAVAIL      0     0     0  cannot open

  pool: rpool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c1t2d0s0  ONLINE       0     0     0
            c1t3d0s0  ONLINE       0     0     0

errors: No known data errors

# zpool create data3 c3t0d1
invalid vdev specification
use ''-f'' to override the following errors:
/dev/dsk/c2t0d0s2 is part of active ZFS pool data. Please see zpool(1M).
/dev/dsk/c2t0d0s8 is part of active ZFS pool data. Please see zpool(1M).
#

huh?  what does c2t0d0* have to do with my attempt to create a pool on
c3t0d1?

-frank

On January 30, 2009 10:03:42 AM -0800 Frank Cusack <fcusack at
fcusack.com>
wrote:
># zpool create data3 c3t0d1
> invalid vdev specification
> use ''-f'' to override the following errors:
> /dev/dsk/c2t0d0s2 is part of active ZFS pool data. Please see zpool(1M).
> /dev/dsk/c2t0d0s8 is part of active ZFS pool data. Please see zpool(1M).
>#
>
> huh?  what does c2t0d0* have to do with my attempt to create a pool on
> c3t0d1?
same error when i tried a different disk, c2t0d1.  i ended up physically
removing c2t0d0 from the system, creating the new pool, and then adding
c2t0d0 back ... booted without errors.

i wasn''t about to try -f for fear of what might happen to c2t0d0.

later on when i am done with the new pool (it''s temporary space) i will
destroy it and try to recreate it and see if i get the same error.  i
am still looking for the answer to this if anyone has any ideas or wants
me to collect any debug data.

-frank

On January 30, 2009 4:51:36 PM -0800 Frank Cusack <fcusack at fcusack.com>
wrote:
> later on when i am done with the new pool (it''s temporary space) i
will
> destroy it and try to recreate it and see if i get the same error.
yup.  this time i couldn''t attach.

# zpool status | grep c.t.d.
          c2t0d0    ONLINE       0     0     0
          c3t0d1    ONLINE       0     0     0
            c1t2d0s0  ONLINE       0     0     0
            c1t3d0s0  ONLINE       0     0     0
# rmformat | grep c.t.d.
     1. Logical Node: /dev/rdsk/c3t0d0p0
     2. Logical Node: /dev/rdsk/c2t0d0p0
     3. Logical Node: /dev/rdsk/c3t0d1p0
     4. Logical Node: /dev/rdsk/c0t0d0p0
# zpool attach data c2t0d0 c3t0d0
invalid vdev specification
use ''-f'' to override the following errors:
/dev/dsk/c2t0d0s0 is part of active ZFS pool data. Please see zpool(1M).
# zpool attach -f data c2t0d0 c3t0d0
invalid vdev specification
the following errors must be manually repaired:
/dev/dsk/c2t0d0s0 is part of active ZFS pool data. Please see zpool(1M).
#

again, i fail to see why an error regarding c2t0d0s* has anything to
do with attaching c3t0d0.  also, this time, i couldn''t remove c2t0d0
from the system because that''s the disk i am trying to attach to.

i suspected that the c2t0d0s0 error was coming from data that still
lived on the new disk, maybe it was previously mounted on that device
and zpool was reporting that device in the error instead of the name
of the device it was currently known as.

so i zeroed (with ''dd'') the first 8k blocks.  didn''t
work.

i then found a hint online that maybe the disks didn''t have the
same partitioning.  don''t know why this would matter as zfs should
write a new partition map when i try to use the whole disk, but i
tried using ''fdisk'' (couldn''t use
''format'' because it doesn''t work
with USB drives ... wtf), but was unable to get the same partition
map written -- the new disk (c3t0d0) was always one cylinder short.
''prtvtoc | fmthard'' wouldn''t work because it would
complain that
a partition was not aligned on a cylinder boundary.

so i then removed c2t0d0 (not sure if it was physically or just export)
and created a new pool on c3t0d0, at which time the partition map / label
written by zfs was now consistent with what was on c2t0d0. [interesting[1] 
that fdisk won''t write the same ''full disk'' partition
that ''zpool'' does.]
after destroying the new test pool, re-importing the pool on c2t0d0, still
zpool was complaining about c2t0d0s0 being active.

what finally worked was that i swapped the two disks physically.  on
reboot my existing pool now was on c3t0d0.  i was then able to attach
the new disk (now c2t0d0) without complaint.  UGH.

hopefully there''s a hint in my narrative that will help someone to
reveal what the problem was.

-frank
[1] annoying

On January 30, 2009 9:57:33 PM -0800 Frank Cusack <fcusack at fcusack.com>
wrote:
> what finally worked was that i swapped the two disks physically.  on
> reboot my existing pool now was on c3t0d0.  i was then able to attach
> the new disk (now c2t0d0) without complaint.  UGH.
>
> hopefully there''s a hint in my narrative that will help someone to
> reveal what the problem was.
and if not, it''s an ongoing problem, so i can easily provide debug
info.
my c[23]t0d1 disks both complain about c2t0d0 when i try to create a
pool on them.  the system will be in this state for at least a week.

-frank