Hi everyone, I just joined the list after finding an unanswered message from Ray Van Dolson in the archives. I''m reproducing his question here, as I''m wondering about the same issue and did not find an answer for it anywhere yet. Can anyone shed any light on this subject? -- Original Message -- What are the technical reasons to not have mismatched replication levels? For example, I am creating a zpool with three raidz vdevs. Two with 8 disks and one with only 7. zpool allows me to do this with -f of course, but I can''t find much documentation on why I shouldn''t other than it''s not recommended. I can understand why, perhaps, for situations where you add new vdevs to your pool later and accidentally use some that aren''t redundant to the same degree others are -- you might unknowingly compromise your vpool in that way... But as long as we''re aware, is there any performance other other technical reason I shouldn''t set up my vdevs as I have above? Thanks, Ray -- End of Original Message -- According to the documentation, here: http://docs.sun.com/app/docs/doc/819-5461/gavwn?a=view "(..)The command also warns you about creating a mirrored or RAID-Z pool using devices of different sizes. While this configuration is allowed, mismatched levels of redundancy result in unused space on the larger device(..)" However, when I create a pool with two raidz groups with 7 vdevs each, and two raidz groups with 7 and 8 vdevs each, I do get more space, indicating the extra space in the largest raidz set is available (which I wouldn''t expect to happen based on the statement above): 2 x raidz ( 7 + 8 ) using 1TB disks: backup2.nbg:~ root# zfs list benchpool78 NAME USED AVAIL REFER MOUNTPOINT benchpool 155K 11.5T 31.0K /benchpool78 backup2.nbg:~ root# zpool list benchpool78 NAME SIZE USED AVAIL CAP HEALTH ALTROOT benchpool 13.6T 354K 13.6T 0% ONLINE - 2 x raidz ( 7 + 7 ) using 1TB disks: backup2.nbg:~ root# zfs list benchpool77 NAME USED AVAIL REFER MOUNTPOINT benchpool 117K 10.6T 1.70K /benchpool77 backup2.nbg:~ root# zpool list benchpool NAME SIZE USED AVAIL CAP HEALTH ALTROOT benchpool 12.6T 146K 12.6T 0% ONLINE - So, is there any real reason for not using mismatched replication levels? Is there any performance penalty? Thanks, Eduardo
On Mon, Mar 1, 2010 at 12:58 PM, Eduardo Bragatto <eduardo at bragatto.com>wrote:> Hi everyone, > > I just joined the list after finding an unanswered message from Ray Van > Dolson in the archives. > > I''m reproducing his question here, as I''m wondering about the same issue > and did not find an answer for it anywhere yet. > > Can anyone shed any light on this subject? > > -- Original Message -- > > What are the technical reasons to not have mismatched replication > levels? > > For example, I am creating a zpool with three raidz vdevs. Two with 8 > disks and one with only 7. zpool allows me to do this with -f of > course, but I can''t find much documentation on why I shouldn''t other > than it''s not recommended. > > I can understand why, perhaps, for situations where you add new vdevs > to your pool later and accidentally use some that aren''t redundant to > the same degree others are -- you might unknowingly compromise your > vpool in that way... > > But as long as we''re aware, is there any performance other other > technical reason I shouldn''t set up my vdevs as I have above? > > Thanks, > Ray > > -- End of Original Message -- > > According to the documentation, here: > > http://docs.sun.com/app/docs/doc/819-5461/gavwn?a=view > > "(..)The command also warns you about creating a mirrored or RAID-Z pool > using devices of different sizes. While this configuration is allowed, > mismatched levels of redundancy result in unused space on the larger > device(..)" > > However, when I create a pool with two raidz groups with 7 vdevs each, and > two raidz groups with 7 and 8 vdevs each, I do get more space, indicating > the extra space in the largest raidz set is available (which I wouldn''t > expect to happen based on the statement above): > > 2 x raidz ( 7 + 8 ) using 1TB disks: > > backup2.nbg:~ root# zfs list benchpool78 > NAME USED AVAIL REFER MOUNTPOINT > benchpool 155K 11.5T 31.0K /benchpool78 > backup2.nbg:~ root# zpool list benchpool78 > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > benchpool 13.6T 354K 13.6T 0% ONLINE - > > 2 x raidz ( 7 + 7 ) using 1TB disks: > > backup2.nbg:~ root# zfs list benchpool77 > NAME USED AVAIL REFER MOUNTPOINT > benchpool 117K 10.6T 1.70K /benchpool77 > backup2.nbg:~ root# zpool list benchpool > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > benchpool 12.6T 146K 12.6T 0% ONLINE - > > So, is there any real reason for not using mismatched replication levels? > Is there any performance penalty? > > Thanks, > Eduardo >The primary concern as I understand it is performance. If they''re close in size, it shouldn''t be a big deal, but when you''ve got mismatched rg''s it can cause quite the performance troubleshooting nightmare. It''s the same reason you don''t want to make a pool that has one raid group that''s entirely 15kRPM SAS drives and the other 5400RPM SATA. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100301/3f264174/attachment.html>
On Mar 1, 2010, at 4:04 PM, Tim Cook wrote:> The primary concern as I understand it is performance. If they''re > close in size, it shouldn''t be a big deal, but when you''ve got > mismatched rg''s it can cause quite the performance troubleshooting > nightmare. It''s the same reason you don''t want to make a pool that > has one raid group that''s entirely 15kRPM SAS drives and the other > 5400RPM SATA.Thanks, Tim. I just finished some tests and they ratify exactly what you said. If it''s of anyone else''s interest, here''s a brief description of my experiment. I have setup three different pools using a total of 15 disks: Case #1 - 3 x raidz ( 5 + 5 + 5 disks ) Case #2 - 2 x raidz ( 7 + 7 disks) + 1 spare Case #3 - 2 x raidz ( 7 + 8 disks) As expected, performance was higher in case #1 and worst in case #3. The differences, however, happened only to write operations with files larger than 1GB. Reading speed was the same across all different scenarios. - Eduardo
Eduardo Bragatto wrote:> On Mar 1, 2010, at 4:04 PM, Tim Cook wrote: > >> The primary concern as I understand it is performance. If they''re >> close in size, it shouldn''t be a big deal, but when you''ve got >> mismatched rg''s it can cause quite the performance troubleshooting >> nightmare. It''s the same reason you don''t want to make a pool that >> has one raid group that''s entirely 15kRPM SAS drives and the other >> 5400RPM SATA. > > Thanks, Tim. I just finished some tests and they ratify exactly what > you said. > > If it''s of anyone else''s interest, here''s a brief description of my > experiment. > > I have setup three different pools using a total of 15 disks: > > Case #1 - 3 x raidz ( 5 + 5 + 5 disks ) > Case #2 - 2 x raidz ( 7 + 7 disks) + 1 spare > Case #3 - 2 x raidz ( 7 + 8 disks) > > As expected, performance was higher in case #1 and worst in case #3. > > The differences, however, happened only to write operations with files > larger than 1GB. >You would also see a performance hit when the pool is getting full, the smaller vdev will fill and all the I/O will be on the bigger one. -- Ian.