All, We''re going to start testing ZFS and I had a question about Top Level Devices (TLDs). In Sun''s class, they specifically said not to use more than 9 TLDs due to performance concerns. Our storage admins make LUNs roughly 15G in size -- so how would we make a large pool (1TB) if we''re limited to only 9 TLDs? The Best Practices guide ( http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations) suggests not using 40+ disks in a RAIDZ TLD: "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration with one logical device of 40+ devices. See the sections below for examples of redundant configurations." Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each RAIDZ? Or is the 9 TLDs an old recommendation that has since been changed? Thanks! Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091201/c82c8048/attachment.html>
Hi Chris, If you have 40 or so disks then you would create 5-6 RAIDZ virtual devices of 7-8 disks each, or possibly include two disks for the root pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks for a non-root pool. This configuration guide hasn''t been updated for RAIDZ-3 yet, but you will get some ideas about how to configure a redundant configuration of many disks, here: http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide See ZFS Configuration Example (x4500 with raidz2) Cindy On 12/01/09 09:20, Christopher White wrote:> All, > > We''re going to start testing ZFS and I had a question about Top Level > Devices (TLDs). In Sun''s class, they specifically said not to use more > than 9 TLDs due to performance concerns. Our storage admins make LUNs > roughly 15G in size -- so how would we make a large pool (1TB) if we''re > limited to only 9 TLDs? > > The Best Practices guide ( > http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations > ) suggests not using 40+ disks in a RAIDZ TLD: > > "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration > with one logical device of 40+ devices. See the sections below for > examples of redundant configurations." > > Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each RAIDZ? > > Or is the 9 TLDs an old recommendation that has since been changed? > > Thanks! > > Chris > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Cindy -- Thanks for the link! I see in one of the examples that there are 14 TLDs (all mirrored). Does that mean there are no performance issues with having more than 9 TLDs? In the Sun class I attended, the instructor said to not use more than 9 TLDs, which seems like it could be very limiting, especially in a SAN setting. Like I said, our storage group presents 15G LUNs to use -- so it''d be difficult to keep the TLDs under 9 and have a very large filesystem. Let me know what you think. Thanks! Chris On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen <Cindy.Swearingen at sun.com>wrote:> Hi Chris, > > If you have 40 or so disks then you would create 5-6 RAIDZ virtual > devices of 7-8 disks each, or possibly include two disks for the root > pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks > for a non-root pool. > > This configuration guide hasn''t been updated for RAIDZ-3 yet, but you > will get some ideas about how to configure a redundant configuration > of many disks, here: > > http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide > > See ZFS Configuration Example (x4500 with raidz2) > > Cindy > > > On 12/01/09 09:20, Christopher White wrote: > >> All, >> >> We''re going to start testing ZFS and I had a question about Top Level >> Devices (TLDs). In Sun''s class, they specifically said not to use more than >> 9 TLDs due to performance concerns. Our storage admins make LUNs roughly >> 15G in size -- so how would we make a large pool (1TB) if we''re limited to >> only 9 TLDs? >> >> The Best Practices guide ( >> http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations) suggests not using 40+ disks in a RAIDZ TLD: >> >> "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration >> with one logical device of 40+ devices. See the sections below for examples >> of redundant configurations." >> >> Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each >> RAIDZ? >> >> Or is the 9 TLDs an old recommendation that has since been changed? >> >> Thanks! >> >> Chris >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091201/d8a032ba/attachment.html>
On Dec 1, 2009, at 8:53 AM, Christopher White wrote:> Cindy -- > > Thanks for the link! > > I see in one of the examples that there are 14 TLDs (all mirrored). > Does that mean there are no performance issues with having more than > 9 TLDs? In the Sun class I attended, the instructor said to not use > more than 9 TLDs, which seems like it could be very limiting, > especially in a SAN setting. Like I said, our storage group > presents 15G LUNs to use -- so it''d be difficult to keep the TLDs > under 9 and have a very large filesystem. > > Let me know what you think. Thanks!I''ve never heard of a top-level vdev limit. Are you sure they are not getting confused with the leaf-vdev suggestions in the zpool(1m) man page? -- richard> > Chris > > > > On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen <Cindy.Swearingen at sun.com > > wrote: > Hi Chris, > > If you have 40 or so disks then you would create 5-6 RAIDZ virtual > devices of 7-8 disks each, or possibly include two disks for the root > pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) > disks > for a non-root pool. > > This configuration guide hasn''t been updated for RAIDZ-3 yet, but you > will get some ideas about how to configure a redundant configuration > of many disks, here: > > http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide > > See ZFS Configuration Example (x4500 with raidz2) > > Cindy > > > On 12/01/09 09:20, Christopher White wrote: > All, > > We''re going to start testing ZFS and I had a question about Top > Level Devices (TLDs). In Sun''s class, they specifically said not to > use more than 9 TLDs due to performance concerns. Our storage > admins make LUNs roughly 15G in size -- so how would we make a large > pool (1TB) if we''re limited to only 9 TLDs? > > The Best Practices guide ( http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations > ) suggests not using 40+ disks in a RAIDZ TLD: > > "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored > configuration with one logical device of 40+ devices. See the > sections below for examples of redundant configurations." > > Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each > RAIDZ? > > Or is the 9 TLDs an old recommendation that has since been changed? > > Thanks! > > Chris > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Chris, The TLD terminology is confusing so let''s think about this way: Performance is best when only 3-9 physical devices are included in each mirror or RAIDZ grouping as shown in the configuration guide. Cindy On 12/01/09 09:53, Christopher White wrote:> Cindy -- > > Thanks for the link! > > I see in one of the examples that there are 14 TLDs (all mirrored). > Does that mean there are no performance issues with having more than 9 > TLDs? In the Sun class I attended, the instructor said to not use more > than 9 TLDs, which seems like it could be very limiting, especially in a > SAN setting. Like I said, our storage group presents 15G LUNs to use -- > so it''d be difficult to keep the TLDs under 9 and have a very large > filesystem. > > Let me know what you think. Thanks! > > Chris > > > > On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen > <Cindy.Swearingen at sun.com <mailto:Cindy.Swearingen at sun.com>> wrote: > > Hi Chris, > > If you have 40 or so disks then you would create 5-6 RAIDZ virtual > devices of 7-8 disks each, or possibly include two disks for the root > pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks > for a non-root pool. > > This configuration guide hasn''t been updated for RAIDZ-3 yet, but you > will get some ideas about how to configure a redundant configuration > of many disks, here: > > http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide > > See ZFS Configuration Example (x4500 with raidz2) > > Cindy > > > On 12/01/09 09:20, Christopher White wrote: > > All, > > We''re going to start testing ZFS and I had a question about Top > Level Devices (TLDs). In Sun''s class, they specifically said > not to use more than 9 TLDs due to performance concerns. Our > storage admins make LUNs roughly 15G in size -- so how would we > make a large pool (1TB) if we''re limited to only 9 TLDs? > > The Best Practices guide ( > http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations > ) suggests not using 40+ disks in a RAIDZ TLD: > > "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored > configuration with one logical device of 40+ devices. See the > sections below for examples of redundant configurations." > > Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in > each RAIDZ? > > Or is the 9 TLDs an old recommendation that has since been changed? > > Thanks! > > Chris > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >
On Tue, 1 Dec 2009, Cindy Swearingen wrote:> The TLD terminology is confusing so let''s think about this way: > > Performance is best when only 3-9 physical devices are included in > each mirror or RAIDZ grouping as shown in the configuration guide.It seems that these "TLDs" are what the rest of us call "vdev"s. Performance improves with more "vdev"s since zfs load-shares across them. The load-sharing helps defend against performance loss due to poorly-performing devices. Each write to a vdev produces one I/O operation to each of the devices in the vdev. While adding more devices to raidz1/raidz2 vdevs increases storage capacity, the increase is at a loss to the number of IOPS which can be sustained. Variability in device performance allows a poorly-performing device to dominate the performance of the whole vdev, and particulary for raidz1/raidz2, which need to be written to synchronously. The loss of IOPS, and the risk of performance loss due to imperfectly-matched hardware, results in increased risk of performance loss with too many devices in a raidz1/raidz2 vdev. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Bob - thanks, that makes sense. The classroom book refers to "top-level virtual devices," and were referred to as TLDs throughout the class (Top-Level Devices). As you noted, those are either the base LUN, mirror, raidz, or raidz2. So there''s no limit to the number of TLDs/vdevs we can have, the only recommendation is that we have no more than 3-9 LUNs per RAIDZ vdev? Chris On Tue, Dec 1, 2009 at 12:25 PM, Bob Friesenhahn < bfriesen at simple.dallas.tx.us> wrote:> On Tue, 1 Dec 2009, Cindy Swearingen wrote: > > The TLD terminology is confusing so let''s think about this way: >> >> Performance is best when only 3-9 physical devices are included in >> each mirror or RAIDZ grouping as shown in the configuration guide. >> > > It seems that these "TLDs" are what the rest of us call "vdev"s. > > Performance improves with more "vdev"s since zfs load-shares across them. > The load-sharing helps defend against performance loss due to > poorly-performing devices. > > Each write to a vdev produces one I/O operation to each of the devices in > the vdev. While adding more devices to raidz1/raidz2 vdevs increases > storage capacity, the increase is at a loss to the number of IOPS which can > be sustained. Variability in device performance allows a poorly-performing > device to dominate the performance of the whole vdev, and particulary for > raidz1/raidz2, which need to be written to synchronously. The loss of IOPS, > and the risk of performance loss due to imperfectly-matched hardware, > results in increased risk of performance loss with too many devices in a > raidz1/raidz2 vdev. > > Bob > -- > Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091201/c1dba65b/attachment.html>
On Tue, 1 Dec 2009, Christopher White wrote:> > So there''s no limit to the number of TLDs/vdevs we can have, the only recommendation is > that we have no more than 3-9 LUNs per RAIDZ vdev?Yes. No one here has complained about problems due to too many vdevs. They have complained due to too many pools on one system, too many filesystems in a pool, and too many disks in one raidz2 vdef. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Hi Chris, It sounds like there is some confusion with the recommendation about raidz? vdevs. It is recommended that each raidz? TLD be a "single-digit" number of disks - so up to 9. The total number of these single digit TLDs is not practically limited. Craig Christopher White wrote:> Cindy -- > > Thanks for the link! > > I see in one of the examples that there are 14 TLDs (all mirrored). Does > that mean there are no performance issues with having more than 9 TLDs? In > the Sun class I attended, the instructor said to not use more than 9 TLDs, > which seems like it could be very limiting, especially in a SAN setting. > Like I said, our storage group presents 15G LUNs to use -- so it''d be > difficult to keep the TLDs under 9 and have a very large filesystem. > > Let me know what you think. Thanks! > > Chris > > > > On Tue, Dec 1, 2009 at 10:47 AM, Cindy Swearingen > <Cindy.Swearingen at sun.com>wrote: > >> Hi Chris, >> >> If you have 40 or so disks then you would create 5-6 RAIDZ virtual >> devices of 7-8 disks each, or possibly include two disks for the root >> pool, two disks as spares, and then 36 (4 RAIDZ vdevs of 6 disks) disks >> for a non-root pool. >> >> This configuration guide hasn''t been updated for RAIDZ-3 yet, but you >> will get some ideas about how to configure a redundant configuration >> of many disks, here: >> >> http://www.solarisinternals.com/wiki/index.php/ZFS_Configuration_Guide >> >> See ZFS Configuration Example (x4500 with raidz2) >> >> Cindy >> >> >> On 12/01/09 09:20, Christopher White wrote: >> >>> All, >>> >>> We''re going to start testing ZFS and I had a question about Top Level >>> Devices (TLDs). In Sun''s class, they specifically said not to use more >>> than >>> 9 TLDs due to performance concerns. Our storage admins make LUNs roughly >>> 15G in size -- so how would we make a large pool (1TB) if we''re limited to >>> only 9 TLDs? >>> >>> The Best Practices guide ( >>> http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Storage_Pool_Performance_Considerations) >>> suggests not using 40+ disks in a RAIDZ TLD: >>> >>> "Avoid creating a RAIDZ, RAIDZ-2, RAIDZ-3, or a mirrored configuration >>> with one logical device of 40+ devices. See the sections below for examples >>> of redundant configurations." >>> >>> Does that mean we should only have 9 RAIDZ TLDs with 39 LUNs in each >>> RAIDZ? >>> >>> Or is the 9 TLDs an old recommendation that has since been changed? >>> >>> Thanks! >>> >>> Chris >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Craig Cory Senior Instructor :: ExitCertified : Sun Certified System Administrator : Sun Certified Network Administrator : Sun Certified Security Administrator : Veritas Certified Instructor 8950 Cal Center Drive Bldg 1, Suite 110 Sacramento, California 95826 [e] craig.cory at exitcertified.com [p] 916.669.3970 [f] 916.669.3977 [w] WWW.EXITCERTIFIED.COM +---------------------------------------------------------------------+ OTTAWA | SACRAMENTO | MONTREAL | LAS VEGAS | QUEBEC CITY | CALGARY SAN FRANCISCO | VANCOUVER | REGINA | WINNIPEG | TORONTO