I have a scenario (tray failure) that I am trying to "predict" how zfs
will behave and am looking for some input . Coming from the world of
svm, ZFS is WAY different ;)
If we have 2 racks, containing 4 trays each, 2 6540''s that present 8D
Raid5 luns to the OS/zfs and through zfs we setup a mirror config such
that: I''m oversimplifying here but...
Rack 1 - Tray 1 = lun 0 Rack 2 - Tray 1 = lun 4
Rack 1 - Tray 2 = lun 1 Rack 2 - Tray 2 = lun 5
Rack 1 - Tray 3 = lun 2 Rack 2 - Tray 3 = lun 6
Rack 1 - Tray 4 = lun 3 Rack 2 - Tray 4 = lun 7
so the zpool command would be:
zpool create somepool mirror 0 4 mirror 1 5 mirror 2 6 mirror 3 7
<---(just for ease of explanation using the "supposed" lun numbers)
so a status output would look similar to:
somepool
mirror
0
4
mirror
1
5
mirror
3
6
mirror
4
7
Now in the VERY unlikely event that we lost the first tray in each rack
which contain 0 and 4 respectively...
somepool
mirror ---
0 |
4 | Bye Bye
---
mirror
1
5
mirror
3
6
mirror
4
7
Would the entire "somepool" zpool die? Would it affect ALL users in
this pool or a portion of the users? Is there a way in zfs to be able
to tell what individual users are hosed (my group is a bunch of control
freaks ;)? How would zfs react to something like this? Also any
feedback on a better way to do this is more then welcome
Please keep in mind I am a "ZFS noob" so detailed explanations would
be
awesome.
Thanks in advance
Robb
On Thu, 10 Jul 2008, Robb Snavely wrote:> > Now in the VERY unlikely event that we lost the first tray in each rack > which contain 0 and 4 respectively... > > somepool > mirror --- > 0 | > 4 | Bye Bye > --- > mirror > 1 > 5 > mirror > 3 > 6 > mirror > 4 > 7 > > > Would the entire "somepool" zpool die? Would it affect ALL users in > this pool or a portion of the users? Is there a way in zfs to be ableZFS loadshares the pool over the VDEVs (mirror is a type of VDEV) so your entire pool would become dead and unusable until the VDEV is restored. You want your mirrors to be based on hardware which is as distinct as possible. If necessary you could consider a tripple mirror. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Essentially yes, the entire pool dies. If you think of each mirror as an individual disk, you''ve just striped them together so the pool goes offline if any mirror fails, and each mirror can only guard against one half of the mirror failing. If you want to guard against any two trays failing, you need to use some kind of dual parity protection. Either dual mirrors, or raid-z2. Given that you only have 8 LUN''s, raid-z2 would seem to be the best option. If you really need to use mirroring for performance, is there any way you can split those trays to generate two LUN''s each? That gives you 16 LUN''s in total, enough for five dual mirror sets (using 3 LUN''s each), plus one acting as a hot spare. This message posted from opensolaris.org
Sorry, but I''m stuck at 6540. There are so many options in how you would practically configure these that there is no way to give a sensible answer to your question. But the most basic questions are: Does the racks have power from separate PDUs? Are they in physically remote locations? Does your fabric switches have redundant power from separate PDUs? Do you want mirroring here purely for performance reasons? Because these systems have so much internal redundancy that I can not see why you would want to mirror across them. Striping would give you better performance. On Thu, Jul 10, 2008 at 11:01 PM, Robb Snavely <Robb.Snavely at sun.com> wrote:> I have a scenario (tray failure) that I am trying to "predict" how zfs > will behave and am looking for some input . Coming from the world of > svm, ZFS is WAY different ;) > > If we have 2 racks, containing 4 trays each, 2 6540''s that present 8D > Raid5 luns to the OS/zfs and through zfs we setup a mirror config such > that: I''m oversimplifying here but... > > Rack 1 - Tray 1 = lun 0 Rack 2 - Tray 1 = lun 4 > Rack 1 - Tray 2 = lun 1 Rack 2 - Tray 2 = lun 5 > Rack 1 - Tray 3 = lun 2 Rack 2 - Tray 3 = lun 6 > Rack 1 - Tray 4 = lun 3 Rack 2 - Tray 4 = lun 7 > > so the zpool command would be: > > zpool create somepool mirror 0 4 mirror 1 5 mirror 2 6 mirror 3 7 > <---(just for ease of explanation using the "supposed" lun numbers) > > so a status output would look similar to: > > somepool > mirror > 0 > 4 > mirror > 1 > 5 > mirror > 3 > 6 > mirror > 4 > 7 > > Now in the VERY unlikely event that we lost the first tray in each rack > which contain 0 and 4 respectively... > > somepool > mirror --- > 0 | > 4 | Bye Bye > --- > mirror > 1 > 5 > mirror > 3 > 6 > mirror > 4 > 7 > > > Would the entire "somepool" zpool die? Would it affect ALL users in > this pool or a portion of the users? Is there a way in zfs to be able > to tell what individual users are hosed (my group is a bunch of control > freaks ;)? How would zfs react to something like this? Also any > feedback on a better way to do this is more then welcome > > Please keep in mind I am a "ZFS noob" so detailed explanations would be > awesome. > > Thanks in advance > > Robb > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Any sufficiently advanced technology is indistinguishable from magic. Arthur C. Clarke Afrikaanse Stap Website: http://www.bloukous.co.za My blog: http://initialprogramload.blogspot.com ICQ = 193944626, YahooIM = johan_hartzenberg, GoogleTalk jhartzen at gmail.com, AIM = JohanHartzenberg -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080711/f6990310/attachment.html>
On Fri, 11 Jul 2008, Ross wrote:> > If you want to guard against any two trays failing, you need to use > some kind of dual parity protection. Either dual mirrors, or > raid-z2. Given that you only have 8 LUN''s, raid-z2 would seem to be > the best option.System reliability will be dominated by the reliability of the weakest VDEV. If all the VDEVs have the same reliability then the reliability of the entire load-shared pool will be the reliability of one VDEV divided by the number of VDEVs. Given sufficient individual VDEV reliability, it can be seen that it takes quite a lot of VDEVs in the load-shared pool before the number of VDEVs becomes very significant in the pool reliability calculation. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Thank you for all the feedback! It''s appreciated! @hartz "Does the racks have power from separate PDUs?" Yes "Are they in physically remote locations?" No, the racks are side by side "Does your fabric switches have redundant power from separate PDUs?" yes "Do you want mirroring here purely for performance reasons?" Goals would be data integrity, and REDUNDANCY - while not throwing performance completely out of the window. "Because these systems have so much internal redundancy that I can not see why you would want to mirror across them." This is due to the fact that we have data that we just can''t afford to lose. Our hardware/power setup is pretty good and we have good backups but want to "try" to make sure all of our customers data is protected from every angle...and as I said in the initial post I know this scenario is possible but not probable especially with the redundant power etc....so in short, mirroring was put in there to account for one of the racks failing (again, unlikely in our setup.."hope for the best...prepare for the worst") "Striping would give you better performance." So how would this be setup in ZFS zpool create somepool 0 1 2 3 4 5 6 7 So essentially a raid 0 on the zfs side? and leave all the redundancy on the hardware?....<shakes nervously> Am I understanding that correctly? This message posted from opensolaris.org