Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I''ve found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
Roy Sigurd Karlsbakk
2011-Mar-04 13:24 UTC
[zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
----- Original Message -----> Hi all > > I just did a small test on RAIDz2 to check whether my suspicion was > right about ZFS not treating spares as replicas/copies of drives, and > I think I''ve found it true. The short story: If two spares replaces > two drives in raidz2, losing a third drive, even with the spares > active, makes the pool unavailable. See full report onUpdate 2010-03-04 14:15 CET I just tested on another system. This one, not in production yet, has a mirrored rpool and a 14-drive RAID10 pool named tos-data. I started a copy from a Windows machine into this CIFS share just to generate some traffic. Then I did a zfs detach of one side of each of the mirrors for tos-data and created a new 5-drive raidz2 pool name jalla with two dedicated spares. I started a dd to fill it up and plugged one drive, waited for it to resilver and plugged another, again waited for the resilver to finish and plugged the third. The server now hangs on all pools. I''ve also tested removing drives from mirrors and waiting for them to resilver to spares. This seems to work as expected, although I doubt booting from one will work without grub being installed.> ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt > PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdfThese are mow updated as well Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
Eric Schrock
2011-Mar-04 16:50 UTC
[zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org. On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don''t necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it''s left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then ''zpool status'' would think your pool is perfectly healthy. In this case it''s true from a data protection standpoint, but not necessarily from a "all is well in the world" standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk <roy at karlsbakk.net>wrote:> Hi all > > I just did a small test on RAIDz2 to check whether my suspicion was right > about ZFS not treating spares as replicas/copies of drives, and I think I''ve > found it true. The short story: If two spares replaces two drives in raidz2, > losing a third drive, even with the spares active, makes the pool > unavailable. See full report on > > ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt > PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf > > Vennlige hilsener / Best regards > > roy > -- > Roy Sigurd Karlsbakk > (+47) 97542685 > roy at karlsbakk.net > http://blogg.karlsbakk.net/ > -- > I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det > er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av > idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate > og relevante synonymer p? norsk. > > _______________________________________________ > Developer mailing list > Developer at lists.illumos.org > http://lists.illumos.org/m/listinfo/developer >-- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110304/03dc4de9/attachment-0001.html>
Roy Sigurd Karlsbakk
2011-Mar-04 16:58 UTC
[zfs-discuss] [illumos-Developer] ZFS spare disk usage issue
I understand that some of it may be a simple bug, but should it hang _all_ the pools? That''s what happens when the third drive is removed... roy ----- Original Message ----- This looks like a pretty simple bug. The issue is that the state of the SPARE vdev is being reported as REMOVED instead of DEGRADED. If it were the latter (as it should be), then everything would work just fine. Please file a bug at bugs.illumos.org . On a side note, this continues to expose the overly simplistic vdev state model used by ZFS (one which I can take a bulk of the responsibility for). Back before the days of ditto blocks and SPA3.0, it was sufficient to model state as a fairly binary proposition. But this now has ramifications that don''t necessarily make sense. For example, one may be able open a pool even if a toplevel vdev is faulted. And even when a spare has finished resilvering, it''s left in the DEGRADED state, which has implications for allocation policies (though I remember discussions around changing this). But the pool state is derived directly from the toplevel vdev state, so if you switch spares to be ONLINE, then ''zpool status'' would think your pool is perfectly healthy. In this case it''s true from a data protection standpoint, but not necessarily from a "all is well in the world" standpoint, as you are down one spare, and that spare may not have the same RAS properties as other devices in your RAID-Z stripe (it may put 3 disks on the same controller in one stripe, for example). - Eric On Fri, Mar 4, 2011 at 7:06 AM, Roy Sigurd Karlsbakk < roy at karlsbakk.net > wrote: Hi all I just did a small test on RAIDz2 to check whether my suspicion was right about ZFS not treating spares as replicas/copies of drives, and I think I''ve found it true. The short story: If two spares replaces two drives in raidz2, losing a third drive, even with the spares active, makes the pool unavailable. See full report on ODT: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.odt PDF: http://karlsbakk.net/ZFS/ZFS%20Spare%20disk%20usage.pdf Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk. _______________________________________________ Developer mailing list Developer at lists.illumos.org http://lists.illumos.org/m/listinfo/developer -- Eric Schrock Delphix 275 Middlefield Road, Suite 50 Menlo Park, CA 94025 http://www.delphix.com -- Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110304/7cbf3f00/attachment.html>