zfs discuss - Sep 2009 - Moving volumes to new controller

Hi,

I''m trying to move disks in a zpool from one SATA-kontroller to
another.
Its 16 disks in 4x4 raidz.
Just to see if it could be done, I moved one disk from one raidz over to
the new controller. Server was powered off.
After booting OS, I get this:
Zpool status
(...)
raidz1     DEGRADED               0     0     0
            c10t4d0  ONLINE       0     0     0
            c10t5d0  ONLINE       0     0     0
            c10t6d0  ONLINE       0     0     0
            c10t7d0  FAULTED      0     0     0  corrupted data
(...)

This looks correct. The c10 controller doesn''t have a t7 anymore.
When I look in /dev/dsk, I can see the "new" disk as c11t0d0, so I try
this:
zpool replace storage c10t7d0 c11t0d0
/dev/dsk/c11t0d0s0 is part of active ZFS pool storage. Please see
zpool(1M).

So it looks like the data on the disk is intact, since zpool clearly
thinks the drive is still in the pool, even if its listed with a new
name..

I''ve tried several things now (fumbling around in the dark :-)).
I tried to delete all partitions and relabel the disk, with no other
results than above.
I can "online" it, but it goes in "faulted state"..

How can I get zpool to realize that c11t0d0 is really c10t7d0? I don''t
have any important data on this array, but I need to know that it can be
fixed if something should happen after I''ve filled it with real data.
So
I don''t really want to destroy it, and make a new pool.
If I need to destroy the data on the one drive, its no problem. I will
probably still be able to resilver the raidz after "replacing" the
drive.
So if there is a way to clear the zpool config from the drive, it will
solve my problem too.

Both controllers are "raid controllers", and I haven''t found
any way to
make them presents the disks directly to opensolaris. So I  have made 1
volume for each drive (the raid5 implementation is rather slow, and they
have no battery). Maybe this is the source of the problems?


-- 
Vidar

Vidar.Nilsen at palantir.no said:
> I''m trying to move disks in a zpool from one SATA-kontroller to
another. Its
> 16 disks in 4x4 raidz. Just to see if it could be done, I moved one disk
from
> one raidz over to the new controller. Server was powered off. 
> . . .
> zpool replace storage c10t7d0 c11t0d0
> /dev/dsk/c11t0d0s0 is part of active ZFS pool storage. Please see
zpool(1M).
> . . .
> I''ve tried several things now (fumbling around in the dark :-)). I
tried to
> delete all partitions and relabel the disk, with no other results than
above.
> . . .
To recover from this situation, you''ll need to erase enough blocks of
the disk to get rid of the ZFS pool info.  You could do this a number of
ways, but probably the simplest is:
	dd if=/dev/zero of=/dev/rdsk/c11t0d0 bs=512 count=100

You may also need to give the same treatment to the last several blocks
of the disk, where redundant ZFS labels may still be present.

> Both controllers are "raid controllers", and I haven''t
found any way to make
> them presents the disks directly to opensolaris. So I  have made 1 volume
for
> each drive (the raid5 implementation is rather slow, and they have no
> battery). Maybe this is the source of the problems? 
I don''t think so.  If the two RAID controllers were not compatible, I
doubt
that ZFS would see the pool info on the disk that you already moved.

By the way, after you''ve got the above issue fixed, if you can power
the
server off, you might be able to move all the drives at once, without any
resilvering.  Just "zpool export storage" before moving the drives,
then
afterwards "zpool import" should be able to find the pool in the new
location.

Note that this export/import approach probably won''t work if the two
RAID
controllers are not compatible with each other.  Some RAID controllers can
be re-flashed with non-RAID firmware, so that might simplify things.

Regards,

Marion

On Thursday, September 17, 2009, Nilsen, Vidar <Vidar.Nilsen at
palantir.no> wrote:
> Hi,
>
> I''m trying to move disks in a zpool from one SATA-kontroller to
another.
> Its 16 disks in 4x4 raidz.
> Just to see if it could be done, I moved one disk from one raidz over to
> the new controller. Server was powered off.
> After booting OS, I get this:
> Zpool status
> (...)
> raidz1 ? ? DEGRADED ? ? ? ? ? ? ? 0 ? ? 0 ? ? 0
>  ? ? ? ? ? ?c10t4d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
>  ? ? ? ? ? ?c10t5d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
>  ? ? ? ? ? ?c10t6d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
>  ? ? ? ? ? ?c10t7d0 ?FAULTED ? ? ?0 ? ? 0 ? ? 0 ?corrupted data
> (...)
>
> This looks correct. The c10 controller doesn''t have a t7 anymore.
> When I look in /dev/dsk, I can see the "new" disk as c11t0d0, so
I try
> this:
> zpool replace storage c10t7d0 c11t0d0
> /dev/dsk/c11t0d0s0 is part of active ZFS pool storage. Please see
> zpool(1M).
>
> So it looks like the data on the disk is intact, since zpool clearly
> thinks the drive is still in the pool, even if its listed with a new
> name..
>
> I''ve tried several things now (fumbling around in the dark :-)).
> I tried to delete all partitions and relabel the disk, with no other
> results than above.
> I can "online" it, but it goes in "faulted state"..
>
> How can I get zpool to realize that c11t0d0 is really c10t7d0? I
don''t
> have any important data on this array, but I need to know that it can be
> fixed if something should happen after I''ve filled it with real
data. So
> I don''t really want to destroy it, and make a new pool.
> If I need to destroy the data on the one drive, its no problem. I will
> probably still be able to resilver the raidz after "replacing"
the
> drive.
> So if there is a way to clear the zpool config from the drive, it will
> solve my problem too.
>
> Both controllers are "raid controllers", and I haven''t
found any way to
> make them presents the disks directly to opensolaris. So I ?have made 1
> volume for each drive (the raid5 implementation is rather slow, and they
> have no battery). Maybe this is the source of the problems?
>
>
> --
> Vidar
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
All you should have had to do was zpool export then zpool import. It
may or may not work still depending on how badly you''ve mangled the
disk

Marion Hakanson said
>
> To recover from this situation, you''ll need to erase enough blocks
of
> the disk to get rid of the ZFS pool info.  You could do this a number
of
> ways, but probably the simplest is:
>        dd if=/dev/zero of=/dev/rdsk/c11t0d0 bs=512 count=100
>
> You may also need to give the same treatment to the last several
blocks
> of the disk, where redundant ZFS labels may still be present.
zpool replace worked fine after dd''ing away begining and end of drive.
> I don''t think so.  If the two RAID controllers were not
compatible, I
> doubt
> that ZFS would see the pool info on the disk that you already moved.
Agree. The new controller listed it as "Foreign volume", but it was
recognized and the data was obviously there.
> By the way, after you''ve got the above issue fixed, if you can
power
the
> server off, you might be able to move all the drives at once, without
> any
> resilvering.  Just "zpool export storage" before moving the
drives,
then
> afterwards "zpool import" should be able to find the pool in the
new
> location.
I dont have physical access to the box right now, 
I''ll try export, recabling and import the rest of the disks tomorrow.
 
>
> Note that this export/import approach probably won''t work if the
two
> RAID
> controllers are not compatible with each other.  Some RAID controllers
> can
> be re-flashed with non-RAID firmware, so that might simplify things.
I''ve tried to google such firmware. The controller card is an adaptec
21610sa, and I can only find a few official firmwares from adaptec,
doesnt look like any of them support non-raid mode..

Thanks alot for the dd tip. 

regards
Vidar