I would just like to confirm or not whether a vdev failure would lead to failure of the whole pool or not. For example, if I created a pool from two RAID-Z2 vdevs, and three drives fail within the first vdev, is all the data within the whole pool unrecoverable? -- This message posted from opensolaris.org
On 10/18/10 06:28 AM, Simon Breden wrote:> I would just like to confirm or not whether a vdev failure would lead to failure of the whole pool or not. > > For example, if I created a pool from two RAID-Z2 vdevs, and three drives fail within the first vdev, is all the data within the whole pool unrecoverable? >Yes. It would be the same a losing a drive in a stripe of two drives. -- Ian.
OK, thanks Ian. Another example: Would you lose all pool data if you had two vdevs: (1) a RAID-Z2 vdev and (2) a two drive mirror vdev, and three drives in the RAID-Z2 vdev failed? -- This message posted from opensolaris.org
On Sun, Oct 17, 2010 at 12:31 PM, Simon Breden <sbreden at gmail.com> wrote:> OK, thanks Ian. > > Another example: > > Would you lose all pool data if you had two vdevs: (1) a RAID-Z2 vdev and (2) a two drive mirror vdev, and three drives in the RAID-Z2 vdev failed?If you lose 1 vdev, you lose the pool. Doesn''t matter what the redundancy level used in the vdev, if you lose more drives than the vdev can handle, you lose the whole pool. An easy way to remember this: - think of vdevs as harddrives - think of the pool as a RAID-0 stripeset Thus, if 1 drive dies, everything in the RAID-0 is lost. Similar for the pool. -- Freddie Cash fjwcash at gmail.com
OK, thanks Freddie, that''s pretty clear. Cheers, Simon -- This message posted from opensolaris.org
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Freddie Cash > > If you lose 1 vdev, you lose the pool.As long as 1 vdev is striped and not mirrored, that''s true. You can only afford to lose a vdev, if your vdev itself is mirrored. You could, for example, create 2 vdev''s of raidz2, and instead of striping them together, mirror them. Then you could lose one of the raidz2''s, and the other half of the mirror is still up.
On Mon, Oct 18, 2010 at 2:34 PM, Edward Ned Harvey <shill at nedharvey.com> wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Freddie Cash >> >> If you lose 1 vdev, you lose the pool. > > As long as 1 vdev is striped and not mirrored, that''s true. > You can only afford to lose a vdev, if your vdev itself is mirrored. > > You could, for example, create 2 vdev''s of raidz2, and instead of striping > them together, mirror them. ?Then you could lose one of the raidz2''s, and > the other half of the mirror is still up.Can you layer vdevs hierarchically? Last time I looked, you couldn''t do that. Regardless, if you want to survive vdev loss, you probably want to set copies=2 on your data (or 3 if you''re exceptionally paranoid), which could be thought of as another layer of mirroring. (And you can do this just on the datasets you really want to keep safe, you don''t have to do it on the whole pool.) -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
On 18/10/2010 15:12, Peter Tribble wrote:> On Mon, Oct 18, 2010 at 2:34 PM, Edward Ned Harvey<shill at nedharvey.com> wrote: >>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of Freddie Cash >>> >>> If you lose 1 vdev, you lose the pool. >> >> As long as 1 vdev is striped and not mirrored, that''s true. >> You can only afford to lose a vdev, if your vdev itself is mirrored. >> >> You could, for example, create 2 vdev''s of raidz2, and instead of striping >> them together, mirror them. Then you could lose one of the raidz2''s, and >> the other half of the mirror is still up. > > Can you layer vdevs hierarchically? Last time I looked, you couldn''t > do that.You mean like a mirror of raidz or a raidz or mirrors ? No the admin interface doesn''t allow you to do that. -- Darren J Moffat
On Mon, Oct 18, 2010 at 6:34 AM, Edward Ned Harvey <shill at nedharvey.com> wrote:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Freddie Cash >> >> If you lose 1 vdev, you lose the pool. > > As long as 1 vdev is striped and not mirrored, that''s true. > You can only afford to lose a vdev, if your vdev itself is mirrored. > > You could, for example, create 2 vdev''s of raidz2, and instead of striping > them together, mirror them. ?Then you could lose one of the raidz2''s, and > the other half of the mirror is still up.How does one do that? Is it as simple as using "attach" instead of "add" when creating the new vdev? For example: zpool create mypool raidz2 disk0 disk1 disk2 disk3 disk4 disk5 disk6 zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13 Would it work for adding a third raidz2 vdev? -- Freddie Cash fjwcash at gmail.com
On 18/10/2010 16:48, Freddie Cash wrote:> On Mon, Oct 18, 2010 at 6:34 AM, Edward Ned Harvey<shill at nedharvey.com> wrote: >>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of Freddie Cash >>> >>> If you lose 1 vdev, you lose the pool. >> >> As long as 1 vdev is striped and not mirrored, that''s true. >> You can only afford to lose a vdev, if your vdev itself is mirrored. >> >> You could, for example, create 2 vdev''s of raidz2, and instead of striping >> them together, mirror them. Then you could lose one of the raidz2''s, and >> the other half of the mirror is still up. > > How does one do that? Is it as simple as using "attach" instead of > "add" when creating the new vdev? > > For example: > > zpool create mypool raidz2 disk0 disk1 disk2 disk3 disk4 disk5 disk6 > zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13 > > Would it work for adding a third raidz2 vdev?There is no support in ZFS for nested vdevs like this. -- Darren J Moffat
On Mon, Oct 18, 2010 at 8:51 AM, Darren J Moffat <darrenm at opensolaris.org> wrote:> On 18/10/2010 16:48, Freddie Cash wrote: >> >> On Mon, Oct 18, 2010 at 6:34 AM, Edward Ned Harvey<shill at nedharvey.com> >> ?wrote: >>>> >>>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>>> bounces at opensolaris.org] On Behalf Of Freddie Cash >>>> >>>> If you lose 1 vdev, you lose the pool. >>> >>> As long as 1 vdev is striped and not mirrored, that''s true. >>> You can only afford to lose a vdev, if your vdev itself is mirrored. >>> >>> You could, for example, create 2 vdev''s of raidz2, and instead of >>> striping >>> them together, mirror them. ?Then you could lose one of the raidz2''s, and >>> the other half of the mirror is still up. >> >> How does one do that? ?Is it as simple as using "attach" instead of >> "add" when creating the new vdev? >> >> For example: >> >> zpool create mypool raidz2 disk0 disk1 disk2 disk3 ?disk4 ?disk5 ?disk6 >> zpool attach mypool raidz2 disk7 disk8 disk9 disk10 disk11 disk12 disk13 >> >> Would it work for adding a third raidz2 vdev? > > There is no support in ZFS for nested vdevs like this.That''s what I though, but wanted to make sure. Thanks. :) -- Freddie Cash fjwcash at gmail.com
So are we all agreed then, that a vdev failure will cause pool loss ? -- This message posted from opensolaris.org
A workaround is to create two pools each with your 1 vdev, make zvols for each of them and export them via iscsi through the localhost interface. Then make a third mirrored pool out of those two iscsi''ed zvols. -- This message posted from opensolaris.org
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Simon Breden > > So are we all agreed then, that a vdev failure will cause pool loss ?Yes. When I said you could mirror a raidzN vdev, it was based on nothing more credible than assumption backed by suggestion I read somewhere in this list. Now, the most credible answer comes from all the people telling me "you''re wrong, you can''t mirror a raidzN" in this thread. ;-)
On Mon, Oct 18, 2010 at 8:18 PM, Simon Breden <sbreden at gmail.com> wrote:> So are we all agreed then, that a vdev failure will cause pool loss ? > --unless you use copies=2 or 3, in which case your data is still safe for those datasets that have this option set. -- - Tuomas
Tuomas: My understanding is that the "copies" functionality doesn''t guarantee that the extra copies will be kept on a different vdev. So that isn''t entirely true. Unfortunately. On 20 October 2010 07:33, Tuomas Leikola <tuomas.leikola at gmail.com> wrote:> On Mon, Oct 18, 2010 at 8:18 PM, Simon Breden <sbreden at gmail.com> wrote: > > So are we all agreed then, that a vdev failure will cause pool loss ? > > -- > > unless you use copies=2 or 3, in which case your data is still safe > for those datasets that have this option set. > -- > - Tuomas > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20101020/eef9ede5/attachment-0001.html>
On 10/19/10 14:33, Tuomas Leikola wrote:> On Mon, Oct 18, 2010 at 8:18 PM, Simon Breden <sbreden at gmail.com> wrote: >> So are we all agreed then, that a vdev failure will cause pool loss ? >> -- > > unless you use copies=2 or 3, in which case your data is still safe > for those datasets that have this option set.This advice is a little too optimistic. Increasing the copies property value on datasets might help in some failure scenarios, but probably not in more catastrophic failures, such as multiple device or hardware failures. cs
On Oct 19, 2010, at 4:33 PM, Tuomas Leikola <tuomas.leikola at gmail.com> wrote:> On Mon, Oct 18, 2010 at 8:18 PM, Simon Breden <sbreden at gmail.com> wrote: >> So are we all agreed then, that a vdev failure will cause pool loss ? >> -- > > unless you use copies=2 or 3, in which case your data is still safe > for those datasets that have this option set.This doesn''t prevent pool loss in the face of a vdev failure, merely reduces the likelihood of file loss due to block corruption. A loss of a vdev (mirror, raidz or non-redundant disk) means the loss of the pool. -Ross
On Tue, 19 Oct 2010, Cindy Swearingen wrote:>> >> unless you use copies=2 or 3, in which case your data is still safe >> for those datasets that have this option set. > > This advice is a little too optimistic. Increasing the copies property > value on datasets might help in some failure scenarios, but probably not > in more catastrophic failures, such as multiple device or hardware > failures.It is 100% too optimistic. The copies option only duplicates the user data. While zfs already duplicates the metadata (regardless of copies setting), it is not designed to function if a vdev fails. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Wed, Oct 20, 2010 at 3:50 AM, Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> On Tue, 19 Oct 2010, Cindy Swearingen wrote: >>> >>> unless you use copies=2 or 3, in which case your data is still safe >>> for those datasets that have this option set. >> >> This advice is a little too optimistic. Increasing the copies property >> value on datasets might help in some failure scenarios, but probably not >> in more catastrophic failures, such as multiple device or hardware >> failures. > > It is 100% too optimistic. ?The copies option only duplicates the user data. > ?While zfs already duplicates the metadata (regardless of copies setting), > it is not designed to function if a vdev fails.Sorry about that. I thought it was already supported with some --force option. My bad.
Bob Friesenhahn wrote:> On Tue, 19 Oct 2010, Cindy Swearingen wrote: >>> >>> unless you use copies=2 or 3, in which case your data is still safe >>> for those datasets that have this option set. >> >> This advice is a little too optimistic. Increasing the copies property >> value on datasets might help in some failure scenarios, but probably not >> in more catastrophic failures, such as multiple device or hardware >> failures. > > It is 100% too optimistic. The copies option only duplicates the user > data. While zfs already duplicates the metadata (regardless of copies > setting), it is not designed to function if a vdev fails. > > BobSome future filesystem (not zfs as currently implemented) could be designed to handle certain vdev failures where multiple vdevs were used without redundancy at the vdev level. In this scenario, the redundant metadata and user data with copies=2+ would still be accessible by virtue of it having been spread across the vdevs, with at least one copy surviving. Expanding upon this design would allow raw space to be added, with redundancy being set by a ''copies'' parameter. I understand the copies parameter to currently be designed and intended as an extra assurance against failures that affect single blocks but not whole devices. I.e. run ZFS on a laptop with a single hard drive, and use copies=2 to protect against bad sectors but not complete drive failures. I have not tested this, however I imagine that performance is the reason to use copies=2 instead of partitioning/slicing the drive into two halves and mirroring the two halves back together. I also recall seeing something about the copies parameter attempting to spread the copies across different devices, as much as possible.