zfs discuss - Jan 2010 - rpool mirror on zvol, can''t offline and detach

I have a netbook with a small internal ssd as rpool. I have an
external usb HDD with much larger storage, as a separate pool, which
is sometimes attached to the netbook. 

I created a zvol on the external pool, the same size as the internal
ssd, and attached it as a mirror to rpool for backup.  I don''t care so
much about it not being bootable, as long as I can read whatever data
I might need in case of failure or loss. 

The mirror works fine, and resilvers properly and selectively when I
use "zpool offline" and "zpool online" on the zvol
submirror.

I don''t want to have the usb disk attached all the time, nor even to
run with the mirror always active (usb is - just - slower than the
internal ssd). I''d like to be able to move this external disk between
hosts, and potentially repeat the rpool mirror for each, having them
resilver whenever the disk is attached.

However, with the rpool mirror in place, I can''t find a way to
"zpool
export black".  It complains that the poool is busy, because of the
zvol in use.  This happens regardless of whether I have set the zvol
submirror offline.  I expected that, with the subdevice in the offline
state, the zvol would be closed.

Any suggestions?  Is this worth filing as a bug (is the device really
offline)?  Would it work differently if I used a file on the external
pool, instead of a zvol? (I haven''t tried that yet, but don''t
really
expect a difference unless umount -f can help). 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100112/7194cbd4/attachment.bin>

I should have mentioned:
 - opensolaris b130
 - of course I could use partitions on the usb disk, but that''s so much
less flexible.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100112/44a86a54/attachment.bin>

Daniel Carosone wrote:
> However, with the rpool mirror in place, I can''t find a way to
"zpool
> export black".  It complains that the poool is busy, because of the
> zvol in use.  This happens regardless of whether I have set the zvol
> submirror offline.  I expected that, with the subdevice in the offline
> state, the zvol would be closed.
>
>   
How did you set the subdevice in the off line state?  Did you detach the 
device from the mirror?

-- 
Ian.

On Tue, Jan 12, 2010 at 02:38:56PM +1300, Ian Collins
wrote:
> How did you set the subdevice in the off line state? 
# zpool offline rpool /dev/zvol/dsk/....

sorry if that wasn''t clear.
> Did you detach the device from the mirror?
No, because then:
 - it will have to resilver fully on next attach, no quick update
 - it will be marked detached, and harder to import for recovery.  I''m
   not even sure it can be easily recovered, maybe with import -D?

If I do detach the subdevice, of course then the external pool can be
exported - but that''s not helpful for the original goal.  If I
shutdown ungracefully, and boot without the external disk attached,
the expected components are faulted, but again that''s not especially
desirable for normal operations.

Perhaps the new "zpool split" will be better for the second issue, 
but won''t address the first issue.  One of the several reasons for
wanting to use a zvol (or file on zfs) is to be able to clone the
backing store for an "import -f" while keeping the original for
further incremental resilvers. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100112/94373405/attachment.bin>

On Jan 11, 2010, at 4:42 PM, Daniel Carosone wrote:
> I have a netbook with a small internal ssd as rpool. I have an
> external usb HDD with much larger storage, as a separate pool, which
> is sometimes attached to the netbook. 
> 
> I created a zvol on the external pool, the same size as the internal
> ssd, and attached it as a mirror to rpool for backup.  I don''t
care so
> much about it not being bootable, as long as I can read whatever data
> I might need in case of failure or loss. 
IMHO, a split mirror is not as good as a decent backup :-)

At one time, Tim Foster''s scripts had a backup feature where you 
could specify a backup flag on a file system which would automatically
backup to removable media.  This used send/recv which is a clean
way of managing such things.  There has been some talk recently
about the future of that feature, see the zfs-auto-snapshot forum to 
catch up on the conversation.

NB. one reason send/recv in ZFS works like a well-designed split
mirror using some other RAID software is because the same method
is used to send incremental snapshots and resilver mirrors.
> The mirror works fine, and resilvers properly and selectively when I
> use "zpool offline" and "zpool online" on the zvol
submirror.
Not really. If you want to split mirrors for "backup" purposes, then 
you need "zpool split" which recently integrated into b131. It takes 
care of the dangling participles.
 -- richard
> I don''t want to have the usb disk attached all the time, nor even
to
> run with the mirror always active (usb is - just - slower than the
> internal ssd). I''d like to be able to move this external disk
between
> hosts, and potentially repeat the rpool mirror for each, having them
> resilver whenever the disk is attached.
> 
> However, with the rpool mirror in place, I can''t find a way to
"zpool
> export black".  It complains that the poool is busy, because of the
> zvol in use.  This happens regardless of whether I have set the zvol
> submirror offline.  I expected that, with the subdevice in the offline
> state, the zvol would be closed.
> 
> Any suggestions?  Is this worth filing as a bug (is the device really
> offline)?  Would it work differently if I used a file on the external
> pool, instead of a zvol? (I haven''t tried that yet, but
don''t really
> expect a difference unless umount -f can help). 
> 
> --
> Dan.
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Mon, Jan 11, 2010 at 06:03:40PM -0800, Richard Elling
wrote:
> IMHO, a split mirror is not as good as a decent backup :-)
I know.. that was more by way of introduction and background.  It''s
not the only method of backup, but since this disk does get plugged
into the netbook frequently enough it seemed like a useful measure, at
the cost of a couple of setup commands (I hoped).

The technical questions here, however, are "why does zpool offline not
close the device", and/or "what else do I have to do to get the device
closed". 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100112/7b9ec798/attachment.bin>

Hi Dan,

I''m not sure I''m following everything here but I will try:

1. How do you offline a zvol? Can you show your syntax?

You can only offline a redundant pool component, such as a file, slice,
or whole disk.

2. What component does "black" represent? Only a pool can be exported.

3. In general, a root pool will always be busy. You can''t unmount it
or export it:

# zpool export rpool
cannot unmount ''/'': Device busy

I would recommend mirroring your root pool as we describe in the
documentation.

Thanks,

Cindy


On 01/11/10 17:42, Daniel Carosone wrote:
> I have a netbook with a small internal ssd as rpool. I have an
> external usb HDD with much larger storage, as a separate pool, which
> is sometimes attached to the netbook. 
> 
> I created a zvol on the external pool, the same size as the internal
> ssd, and attached it as a mirror to rpool for backup.  I don''t
care so
> much about it not being bootable, as long as I can read whatever data
> I might need in case of failure or loss. 
> 
> The mirror works fine, and resilvers properly and selectively when I
> use "zpool offline" and "zpool online" on the zvol
submirror.
> 
> I don''t want to have the usb disk attached all the time, nor even
to
> run with the mirror always active (usb is - just - slower than the
> internal ssd). I''d like to be able to move this external disk
between
> hosts, and potentially repeat the rpool mirror for each, having them
> resilver whenever the disk is attached.
> 
> However, with the rpool mirror in place, I can''t find a way to
"zpool
> export black".  It complains that the poool is busy, because of the
> zvol in use.  This happens regardless of whether I have set the zvol
> submirror offline.  I expected that, with the subdevice in the offline
> state, the zvol would be closed.
> 
> Any suggestions?  Is this worth filing as a bug (is the device really
> offline)?  Would it work differently if I used a file on the external
> pool, instead of a zvol? (I haven''t tried that yet, but
don''t really
> expect a difference unless umount -f can help). 
> 
> --
> Dan.
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Dan,

I see now how you might have created this config.

I tried to reproduce this issue by creating a separate pool on another
disk and a volume to attach to my root pool, but my system panics when
I try to attach the volume to the root pool.

This is on Nevada, build 130.

Panic aside, we don''t currently recommend building pool components on
top of other pool components because deadlocks can occur.

Cindy


On 01/11/10 19:00, Daniel Carosone wrote:
> On Tue, Jan 12, 2010 at 02:38:56PM +1300, Ian Collins wrote:
>> How did you set the subdevice in the off line state? 
> 
> # zpool offline rpool /dev/zvol/dsk/....
> 
> sorry if that wasn''t clear.
> 
>> Did you detach the device from the mirror?
> 
> No, because then:
>  - it will have to resilver fully on next attach, no quick update
>  - it will be marked detached, and harder to import for recovery. 
I''m
>    not even sure it can be easily recovered, maybe with import -D?
> 
> If I do detach the subdevice, of course then the external pool can be
> exported - but that''s not helpful for the original goal.  If I
> shutdown ungracefully, and boot without the external disk attached,
> the expected components are faulted, but again that''s not
especially
> desirable for normal operations.
> 
> Perhaps the new "zpool split" will be better for the second
issue,
> but won''t address the first issue.  One of the several reasons for
> wanting to use a zvol (or file on zfs) is to be able to clone the
> backing store for an "import -f" while keeping the original for
> further incremental resilvers. 
> 
> --
> Dan.
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Tue, Jan 12, 2010 at 01:26:15PM -0700, Cindy Swearingen
wrote:
> I see now how you might have created this config.
>
> I tried to reproduce this issue by creating a separate pool on another
> disk and a volume to attach to my root pool, but my system panics when
> I try to attach the volume to the root pool.
Oops.. "Now you have two problems".
> This is on Nevada, build 130.
>
> Panic aside, we don''t currently recommend building pool components
on
> top of other pool components because deadlocks can occur.
Aside from all of that... forget it happened to be a zvol.  Say it was
a file on NFS or UFS, or a lofi crypto device, or whatever other kind
of block device, the issue remains: zpool offline doesn''t seem to
cause the device to be closed, leading to various other kinds 
of problems when trying to remove that device or underlying layers. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100113/71cdc750/attachment.bin>