zfs discuss - Feb 2010 - What would happen with a zpool if you ''mirrored'' a disk...

Hi All,

I''ve been using ZFS for a while now - and everything''s been
going well. I
use it under FreeBSD - but this question almost certainly should be the 
same answer, whether it''s FreeBSD or Solaris (I think/hope :)...


Imagine if I have a zpool with 2 disks in it, that are mirrored:

"
NAME         STATE     READ WRITE CKSUM
vol          ONLINE       0     0     0
   mirror    ONLINE       0     0     0
     ad1     ONLINE       0     0     0
     ad2     ONLINE       0     0     0
"

(The device names are FreeBSD disks)

If I offline ''ad2'' - and then did:

"
dd if=/dev/ad1 of=/dev/ad2
"

(i.e. make a mirror copy of ad1 to ad2 - on a *running* system).


What would happen when I tried to ''online'' ad2 again?


I fully expect it might not be pleasant... I''m just curious as to
what''s
going to happen.


When I ''online'' ad2 will ZFS look at it, and be clever enough
to figure out
the disk is obviously corrupt/unusable/has bad meta data on it - and 
resilver accordingly?

Or is it going to see what it thinks is another ''ad1'' and get
a little
upset?


I''m trying to setup something here so I can test what happens - I just 
thought I''d ask around a bit to see if anyone knows what''ll
happen from
past experience.


Thanks,

-Karl

--On 04 February 2010 11:31 +0000 Karl Pielorz <kpielorz_lst at tdx.co.uk>
wrote:
> What would happen when I tried to ''online'' ad2 again?
A reply to my own post... I tried this out, when you make
''ad2'' online
again, ZFS immediately logs a ''vdev corrupt'' failure, and
marks ''ad2''
(which at this point is a byte-for-byte copy of ''ad1'' as it
was being
written to in background) as ''FAULTED'' with
''corrupted data''.

You can''t "replace" it with itself at that point, but a
detach on ad2, and
then attaching ad2 back to ad1 results in a resilver, and recovery.

So to answer my own question - from my tests it looks like you can do this, 
and "get away with it". It''s probably not ideal, but it does
work.

A safer bet would be to detach the drive from the pool, and then re-attach 
it (at which point ZFS assumes it''s a new drive and probably ignores
the
''mirror image'' data that''s on it).

-Karl

(The reason for testing this is because of a weird RAID setup I have where 
if ''ad2'' fails, and gets replaced - the RAID controller is
going to mirror
''ad1'' over to ''ad2'' - and cannot be stopped.
However, once the re-mirroring
is complete the RAID controller steps out the way, and allows raw access to 
each disk in the mirror. Strange, a long story - but true).

Seems your controller is actually doing only harm here, or am I missing
something?

On Feb 4, 2010 8:46 AM, "Karl Pielorz" <kpielorz_lst at
tdx.co.uk> wrote:


--On 04 February 2010 11:31 +0000 Karl Pielorz <kpielorz_lst at tdx.co.uk>
wrote:
> What would happen...
A reply to my own post... I tried this out, when you make
''ad2'' online
again, ZFS immediately logs a ''vdev corrupt'' failure, and
marks ''ad2'' (which
at this point is a byte-for-byte copy of ''ad1'' as it was being
written to in
background) as ''FAULTED'' with ''corrupted
data''.

You can''t "replace" it with itself at that point, but a
detach on ad2, and
then attaching ad2 back to ad1 results in a resilver, and recovery.

So to answer my own question - from my tests it looks like you can do this,
and "get away with it". It''s probably not ideal, but it does
work.

A safer bet would be to detach the drive from the pool, and then re-attach
it (at which point ZFS assumes it''s a new drive and probably ignores
the
''mirror image'' data that''s on it).

-Karl

(The reason for testing this is because of a weird RAID setup I have where
if ''ad2'' fails, and gets replaced - the RAID controller is
going to mirror
''ad1'' over to ''ad2'' - and cannot be stopped.
However, once the re-mirroring
is complete the RAID controller steps out the way, and allows raw access to
each disk in the mirror. Strange, a long story - but true).


_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.or...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100204/1dd9a656/attachment.html>

On Thu, 4 Feb 2010, Karl Pielorz wrote:
> The reason for testing this is because of a weird RAID setup I have 
> where if ''ad2'' fails, and gets replaced - the RAID
controller is going
> to mirror ''ad1'' over to ''ad2'' - and
cannot be stopped.
Does the raid controller not support a JBOD mode?


Regards,
markm

--On 04 February 2010 08:58 -0500 Jacob Ritorto <jacob.ritorto at
gmail.com>
wrote:
> Seems your controller is actually doing only harm here, or am I missing
> something?
The RAID controller presents the drives as both a mirrored pair, and JBOD - 
*at the same time*...

The machine boots off the partition on the ''mirrored'' pair -
and ZFS uses
the JBOD devices (a different area of, of course).

It''s a little weird to say the least - and I wouldn''t
recommend it, but it
does work ''for me'' - and is a way of getting the system to
boot off a
mirror, and still be able to use ZFS with only 2 drives available in the 
chassis.

-Karl

On 04/02/2010 13:45, Karl Pielorz wrote:
>
> --On 04 February 2010 11:31 +0000 Karl Pielorz 
> <kpielorz_lst at tdx.co.uk> wrote:
>
>> What would happen when I tried to ''online'' ad2 again?
>
> A reply to my own post... I tried this out, when you make
''ad2'' online
> again, ZFS immediately logs a ''vdev corrupt'' failure, and
marks ''ad2''
> (which at this point is a byte-for-byte copy of ''ad1'' as
it was being
> written to in background) as ''FAULTED'' with
''corrupted data''.
>
> You can''t "replace" it with itself at that point, but a
detach on ad2,
> and then attaching ad2 back to ad1 results in a resilver, and recovery.
>
> So to answer my own question - from my tests it looks like you can do 
> this, and "get away with it". It''s probably not ideal,
but it does work.
>
it is actually fine - zfs is designed to detect and fix corruption like 
the one you induced.

> A safer bet would be to detach the drive from the pool, and then 
> re-attach it (at which point ZFS assumes it''s a new drive and
probably
> ignores the ''mirror image'' data that''s on it).
>
Yes, it should and if you want to force resynchronization that''s 
probably the best way to do it.
Other thing is that if you suspect some of your data to be corrupted on 
a half of mirror you might try to run zpool scrub as it will fix only 
those corrupted blocks instead of resynchronizing entire mirror which 
might be faster and safer approach.


-- 
Robert Milkowski
http://milek.blogspot.com