zfs discuss - Dec 2009 - Accidentally added disk instead of attaching

Hi,

I wanted to add a disk to the tank pool to create a mirror. I accidentally used
zpool add ? instead of zpool attach ? and now the disk is added. Is there a way
to remove the disk without loosing data? Or maybe change it to mirror?

Thanks,
Martijn
-- 
This message posted from opensolaris.org

> I wanted to add a disk to the tank pool to create a mirror. I accidentally
> used zpool add ? instead of zpool attach ? and now the disk is added. Is
> there a way to remove the disk without loosing data?
Been there, done that -- at a customer site while showing off
ZFS. :-)

Currently, you cannot remove a "simple" device.  Depending
on your Solaris version, you can remove things like hot spares and
cache devices, but not simple vdevs.

Backup the pool and recreate it in the correct way.


Sorry for the bad news -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

I saw this bug report?

http://bugs.opensolaris.org/view_bug.do?bug_id=4852783
-- 
This message posted from opensolaris.org

Do you know a good zfs backup restore walktrough online?
-- 
This message posted from opensolaris.org

> Do you know a good zfs backup restore walktrough online?
Not really.  Check out the zfs send and receive commands in
the ZFS Administration Guide:

http://docs.sun.com/app/docs/doc/819-5461/gbinw?a=view

Basically, you make a snapshot of every filesystem you want to backup,
then you use "zfs send" for each snapshot to create a stream that you 
store in some other place.

Next, you destroy the pool and recreate it.  Finally, you restore
the filesystems one by one with the "zfs receive" command, using the
streams that you created in the first step.

See also the section on send and receive in zfs(1M).


Good luck -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

on 06/12/2009 19:40 Volker A. Brandt said the following:
>> I wanted to add a disk to the tank pool to create a mirror. I
accidentally
>> used zpool add ? instead of zpool attach ? and now the disk is added.
Is
>> there a way to remove the disk without loosing data?
> 
> Been there, done that -- at a customer site while showing off
> ZFS. :-)
> 
> Currently, you cannot remove a "simple" device.  Depending
> on your Solaris version, you can remove things like hot spares and
> cache devices, but not simple vdevs.
> 
> Backup the pool and recreate it in the correct way.
> 
> 
> Sorry for the bad news -- Volker
Yep.  My 2 cents -- ''add'' and ''attach'' are
so similar the words that I think
that ZFS tools UI designers (if any) should re-consider naming of these
commands.  Or ''add'' command should always be interactive and
ask for at least
two confirmations that a user knows what he is doing and why.  Perhaps, it
should include a ZFS micro-exam too.
Jokes aside, this is too easy to make a mistake with the consequences that are
too hard to correct.  Anyone disagrees?

-- 
Andriy Gapon

+1

I support a replacement for a SCM system that used "open" as an alias
for "edit" and a separate command, "opened" to see what was
opened for edit, delete, etc.  Our customers accidentally used "open"
when they meant "opened" so many times that we blocked it as a
command.  It saved us a lot of support calls.

Gary
-- 
This message posted from opensolaris.org

I agree that zpool attach and add look similar in their syntax,
but if you attempt to "add" a disk to a redundant config,
you''ll
see an error message similar to the following:

# zpool status export
   pool: export
  state: ONLINE
  scrub: none requested
config:

         NAME        STATE     READ WRITE CKSUM
         export      ONLINE       0     0     0
           mirror-0  ONLINE       0     0     0
             c1t3d0  ONLINE       0     0     0
             c1t4d0  ONLINE       0     0     0

errors: No known data errors
# zpool add export c1t6d0
invalid vdev specification
use ''-f'' to override the following errors:
mismatched replication level: pool uses mirror and new vdev is disk

Doesn''t the "mismatched replication" message help?

Cindy

On 12/06/09 14:13, Andriy Gapon wrote:
> on 06/12/2009 19:40 Volker A. Brandt said the following:
>>> I wanted to add a disk to the tank pool to create a mirror. I
accidentally
>>> used zpool add ? instead of zpool attach ? and now the disk is
added. Is
>>> there a way to remove the disk without loosing data?
>> Been there, done that -- at a customer site while showing off
>> ZFS. :-)
>>
>> Currently, you cannot remove a "simple" device.  Depending
>> on your Solaris version, you can remove things like hot spares and
>> cache devices, but not simple vdevs.
>>
>> Backup the pool and recreate it in the correct way.
>>
>>
>> Sorry for the bad news -- Volker
> 
> Yep.  My 2 cents -- ''add'' and ''attach''
are so similar the words that I think
> that ZFS tools UI designers (if any) should re-consider naming of these
> commands.  Or ''add'' command should always be interactive
and ask for at least
> two confirmations that a user knows what he is doing and why.  Perhaps, it
> should include a ZFS micro-exam too.
> Jokes aside, this is too easy to make a mistake with the consequences that
are
> too hard to correct.  Anyone disagrees?
>

On Dec 7, 2009, at 10:42 AM, Cindy Swearingen wrote:
> I agree that zpool attach and add look similar in their syntax,
> but if you attempt to "add" a disk to a redundant config,
you''ll
> see an error message similar to the following:
>
> # zpool status export
>  pool: export
> state: ONLINE
> scrub: none requested
> config:
>
>        NAME        STATE     READ WRITE CKSUM
>        export      ONLINE       0     0     0
>          mirror-0  ONLINE       0     0     0
>            c1t3d0  ONLINE       0     0     0
>            c1t4d0  ONLINE       0     0     0
>
> errors: No known data errors
> # zpool add export c1t6d0
> invalid vdev specification
> use ''-f'' to override the following errors:
> mismatched replication level: pool uses mirror and new vdev is disk
This was one of the first, great debates on ZFS-discuss
http://www.opensolaris.org/jive/thread.jspa?messageID=15363&#15363
I believe all possible observations are made in that thread.

Of particular note, humans train themselves to use a "-f" (or for  
kill, "-9" :-)
as a not-optional part of the command string. It really doesn''t matter
what the command is named, you will train yourself to go to the forced
option. If you want to protect yourself from yourself, begin with  
yourself.
-- richard

Andriy Gapon writes:
> on 06/12/2009 19:40 Volker A. Brandt said the following:
> >> I wanted to add a disk to the tank pool to create a mirror. I
accidentally
> >> used zpool add ? instead of zpool attach ? and now the disk is
added. Is
> >> there a way to remove the disk without loosing data?
> >
> > Been there, done that -- at a customer site while showing off
> > ZFS. :-)
[...]
>
> Yep.  My 2 cents -- ''add'' and ''attach''
are so similar the words that I think
> that ZFS tools UI designers (if any) should re-consider naming of these
> commands.  Or ''add'' command should always be interactive
and ask for at least
> two confirmations that a user knows what he is doing and why.  Perhaps, it
> should include a ZFS micro-exam too.
> Jokes aside, this is too easy to make a mistake with the consequences that
are
> too hard to correct.  Anyone disagrees?
I absolutely totally, like, fully agree, man! :-)

There should be some interaction and maybe even a display of the new
pool structure, followed for a confirmation prompt.  This could be
overridden with the already well-established "-f" flag.


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH              WWW: http://www.bb-c.de/~vab/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 45
Gesch?ftsf?hrer: Rainer J. H. Brandt und Volker A. Brandt

> Jokes aside, this is too easy to make a mistake with
> the consequences that are
> too hard to correct.  Anyone disagrees?
No, and this sums up the situation nicely, in that there are two parallel paths
toward a resolution:

 - make the mistake harder to make (various ideas here)
 - make the consequences easier to correct (coming)

We really can and should do both.
-- 
This message posted from opensolaris.org

> but if you attempt to "add" a disk to a redundant
> config, you''ll see an error message similar [..]
> 
> Doesn''t the "mismatched replication" message help?
Not if you''re trying to make a single disk pool redundant by adding ..
er, attaching .. a mirror; then there won''t be such a warning, however
effective that warning might or might not be otherwise.
-- 
This message posted from opensolaris.org

On Dec 7, 2009, at 2:23 PM, Daniel Carosone wrote:
>> but if you attempt to "add" a disk to a redundant
>> config, you''ll see an error message similar [..]
>>
>> Doesn''t the "mismatched replication" message help?
>
> Not if you''re trying to make a single disk pool redundant by  
> adding .. er, attaching .. a mirror; then there won''t be such a  
> warning, however effective that warning might or might not be  
> otherwise.
Not a problem because you can then detach the vdev and add it.
  -- richard

On Dec 7, 2009, at 11:23 PM, Daniel Carosone wrote:
>> but if you attempt to "add" a disk to a redundant
>> config, you''ll see an error message similar [..]
>> 
>> Doesn''t the "mismatched replication" message help?
> 
> Not if you''re trying to make a single disk pool redundant by
adding .. er, attaching .. a mirror; then there won''t be such a
warning, however effective that warning might or might not be otherwise.
That''s exactly where I made the mistake, a single disk pool and I
wanted to attach another disk to build a mirror.

I''m now using zfs send and receive to backup the disk to another zpool.
Then I want to rename the old and the new zpools and mount the new zpool. Then I
can destroy the old zpool, attach the free disks from the old zpool creating a
mirror and in the end detach the extra disk from new mirrored zpool.
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Mon, Dec 7, 2009 at 12:42 PM, Cindy Swearingen
<Cindy.Swearingen at sun.com> wrote:
> I agree that zpool attach and add look similar in their syntax,
> but if you attempt to "add" a disk to a redundant config,
you''ll
> see an error message similar to the following:
>
> # zpool status export
> ?pool: export
> ?state: ONLINE
> ?scrub: none requested
> config:
>
> ? ? ? ?NAME ? ? ? ?STATE ? ? READ WRITE CKSUM
> ? ? ? ?export ? ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ? ?mirror-0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ? ? ?c1t3d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
> ? ? ? ? ? ?c1t4d0 ?ONLINE ? ? ? 0 ? ? 0 ? ? 0
>
> errors: No known data errors
> # zpool add export c1t6d0
> invalid vdev specification
> use ''-f'' to override the following errors:
> mismatched replication level: pool uses mirror and new vdev is disk
>
> Doesn''t the "mismatched replication" message help?
When adding a disk to a single-disk pool, this message isn''t given and
the add proceeds without any warning and without the need to force it:

# cd /tmp
# mkfile 256m f1 f2
# zpool create testpool /tmp/f1
# zpool add testpool /tmp/f2
# zpool status testpool
  pool: testpool
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        testpool    ONLINE       0     0     0
          /tmp/f1   ONLINE       0     0     0
          /tmp/f2   ONLINE       0     0     0

errors: No known data errors

Would it be beneficial to have a command line option to zpool that
would only "preview" or do a "dry-run" through the changes,
but
instead just display what the pool would look like after the operation
and leave the pool unchanged?  For those that very rarely make pool
changes, getting in the habit of always using an option like this
might be a good way to ensure the change is really what is desired.

Some information that might be nice to see would be the before and
after versions of "zpool list", the "zpool status", and what
command
could be run to reverse the change, or a warning if the change is
irreversible like the case with "zpool add".


Ed Plese

On Dec 7, 2009, at 11:32 PM, Ed Plese wrote:
> On Mon, Dec 7, 2009 at 12:42 PM, Cindy Swearingen
> <Cindy.Swearingen at sun.com> wrote:
>> I agree that zpool attach and add look similar in their syntax,
>> but if you attempt to "add" a disk to a redundant config,
you''ll
>> see an error message similar to the following:
>> 
>> # zpool status export
>>  pool: export
>>  state: ONLINE
>>  scrub: none requested
>> config:
>> 
>>        NAME        STATE     READ WRITE CKSUM
>>        export      ONLINE       0     0     0
>>          mirror-0  ONLINE       0     0     0
>>            c1t3d0  ONLINE       0     0     0
>>            c1t4d0  ONLINE       0     0     0
>> 
>> errors: No known data errors
>> # zpool add export c1t6d0
>> invalid vdev specification
>> use ''-f'' to override the following errors:
>> mismatched replication level: pool uses mirror and new vdev is disk
>> 
>> Doesn''t the "mismatched replication" message help?
> 
> When adding a disk to a single-disk pool, this message isn''t given
and
> the add proceeds without any warning and without the need to force it:
> 
> # cd /tmp
> # mkfile 256m f1 f2
> # zpool create testpool /tmp/f1
> # zpool add testpool /tmp/f2
> # zpool status testpool
>  pool: testpool
> state: ONLINE
> scrub: none requested
> config:
> 
>        NAME        STATE     READ WRITE CKSUM
>        testpool    ONLINE       0     0     0
>          /tmp/f1   ONLINE       0     0     0
>          /tmp/f2   ONLINE       0     0     0
> 
> errors: No known data errors
> 
> Would it be beneficial to have a command line option to zpool that
> would only "preview" or do a "dry-run" through the
changes, but
> instead just display what the pool would look like after the operation
> and leave the pool unchanged?  For those that very rarely make pool
> changes, getting in the habit of always using an option like this
> might be a good way to ensure the change is really what is desired.
> 
> Some information that might be nice to see would be the before and
> after versions of "zpool list", the "zpool status", and
what command
> could be run to reverse the change, or a warning if the change is
> irreversible like the case with "zpool add".
I guess the best solution is to make it easy to undo the changes. zfs remove
should be just as easy.
> 
> 
> Ed Plese
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Mon, Dec 7, 2009 at 4:32 PM, Ed Plese <ed at edplese.com>
wrote:
> Would it be beneficial to have a command line option to zpool that
> would only "preview" or do a "dry-run" through the
changes, but
> instead just display what the pool would look like after the operation
> and leave the pool unchanged? ?For those that very rarely make pool
> changes, getting in the habit of always using an option like this
> might be a good way to ensure the change is really what is desired.
There I go requesting features that are already there.  "zpool add"
already has a -n option that will show the pool configuration that
would be used, without actually making any changes.  A couple other
subcommands accept the -n option as well.


Ed Plese

>>> Doesn''t the "mismatched replication" message
help?
>>
>> Not if you''re trying to make a single disk pool redundant by
adding ..
>> er, attaching .. a mirror; then there won''t be such a warning,
however
>> effective that warning might or might not be otherwise.
>
> Not a problem because you can then detach the vdev and add it.
It''s a problem if you''re trying to do that, but end up adding
instead
of attaching, which you can''t (yet) undo.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091208/43bbc2d3/attachment.bin>

What about removing attach/deattach and replace it with
zpool add [-fn] ''pool'' submirror
''device/mirrorname'' ''new_device''
e.g.
        NAME        STATE     READ WRITE CKSUM
        rpool        ONLINE       0     0     0
          mirror-01 ONLINE       0     0     0
            c4d0s0  ONLINE       0     0     0
            c3d0s0  ONLINE       0     0     0
zpool add rpool submirror mirror-01 c5d0s0 # or
zpool add rpool submirror c4d0s0 c5d0s0
zpool remove rpool c5d0s0
Some more examples
zpool add ''pool'' submirror log-01 c7d0s0  # create a mirror
for the Intent Log
And may be one day zpool add ''pool'' subraidz raidz2-01 c5d0s0
to add extra disk to raidz group and have the disk restriped in the background

Which would mean vdev in terms of syntax would support
concat (was disk), concat-file (was file), mirror, submirror, raidz, raidzN,
subraidz (one day), spare, log, cache
--
And change 
zpool add rpool disk c5d0s0
to
zpool add rpool concat c5d0s0  # instead of disk use concat or 
zpool add rpool concatfile <path to file> # instead of file

Cheers
-- 
This message posted from opensolaris.org

Daniel Carosone <dan at geek.com.au> writes:
>>> Not if you''re trying to make a single disk pool redundant
by adding
>>> ..  er, attaching .. a mirror; then there won''t be such a
warning,
>>> however effective that warning might or might not be otherwise.
>>
>> Not a problem because you can then detach the vdev and add it.
>
> It''s a problem if you''re trying to do that, but end up
adding instead
> of attaching, which you can''t (yet) undo.
at least in that case the amount of data shuffling you have to do is
limited to one disk (it''s unlikely you do this mistake for a multi
device vdev).

in any case, the block rewrite implementation isn''t *that* far away, is
it?
-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game

On Mon, 2009-12-07 at 23:31 +0100, Martijn de Munnik
wrote:
> On Dec 7, 2009, at 11:23 PM, Daniel Carosone wrote:
> 
> >> but if you attempt to "add" a disk to a redundant
> >> config, you''ll see an error message similar [..]
> >> 
> >> Doesn''t the "mismatched replication" message
help?
> > 
> > Not if you''re trying to make a single disk pool redundant by
adding .. er, attaching .. a mirror; then there won''t be such a
warning, however effective that warning might or might not be otherwise.
> 
> That''s exactly where I made the mistake, a single disk pool and I
wanted to attach another disk to build a mirror.
> 
> I''m now using zfs send and receive to backup the disk to another
zpool. Then I want to rename the old and the new zpools and mount the new zpool.
Then I can destroy the old zpool, attach the free disks from the old zpool
creating a mirror and in the end detach the extra disk from new mirrored zpool.
Will this be the best way do recreate the mirror with the least
downtime? I can do the zfs send/receive when the system is running. I
only need to do a zfs send/receive of an incremantel snapshot when the
system is not using the zpool (I need to disable services like mail and
db which use zpool for storage). Then I need to rename the zpools and
attach the new zpool and can bring back the services
online.
> 
> > -- 
> > This message posted from opensolaris.org
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> > 
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>