Hi everyone, Here are some current ZFS/DMU benchmark results that might be of interest to you. These results were obtained on a Sunfire x4500 with these specifications: - 2 dual-core AMD Opteron processors - 16 GB main memory - 48 SATA II HDD (7200 RPM, 500 GB each) - Solaris 10 update 3 - Userspace DMU from OpenSolaris build 74 The tool used in this particular benchmark was PIOS, which simulates a parallel I/O load typically experienced by an OSS. Since PIOS links directly with the DMU, it should give a good estimation of the maximum throughput of the userspace DMU-OSS. For comparison, you can also see the results of running the same PIOS benchmark with normal POSIX I/O on the Solaris 10u3 ZFS implementation. See the attached PDF for the results. First page provides PIOS throughput information on striped pools with 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, the kernel ZFS implementation scales better to 24 and 46 disks. Surprisingly enough, disabling checksums has a negative impact on the throughput with a 10-disk striped pool. This impact can be attributed to the differences in the uDMU IO pipeline which are probably causing the IOs to not being parallelized when checksumming is disabled. This issue, however, should not be hard to fix. Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and mirrored pools. In these configurations, the kernel ZFS currently has a better throughput than the userspace DMU. These results were obtained with only minimal tuning of the DMU (setting a maximum cache size and increasing the number of I/O threads). Work is already underway to improve the performance of the DMU and we expect to see better throughput in the coming months. Best regards, Ricardo -- <http://www.sun.com> * Ricardo Manuel Correia * Lustre Engineering Group *Sun Microsystems, Inc.* Portugal Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071113/a7e9a9ee/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Lustre-uDMU.pdf Type: application/pdf Size: 66695 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071113/a7e9a9ee/attachment-0002.pdf
Hi Ricardo, On Tue, Nov 13, 2007 at 11:48:38AM +0000, Ricardo Correia wrote:> Here are some current ZFS/DMU benchmark results that might be of > interest to you.It''s nice to see these results - thanks for posting them. When you have time, could you do a comparison with Lustre 1.6.x on ldiskfs + software RAID 5? It would be good to know how ZFS stands up (probably quite favourably, even at this early stage.) Cheers, Jody> These results were obtained on a Sunfire x4500 with these specifications: > > - 2 dual-core AMD Opteron processors > - 16 GB main memory > - 48 SATA II HDD (7200 RPM, 500 GB each) > - Solaris 10 update 3 > - Userspace DMU from OpenSolaris build 74 > > The tool used in this particular benchmark was PIOS, which simulates a > parallel I/O load typically experienced by an OSS. Since PIOS links > directly with the DMU, it should give a good estimation of the maximum > throughput of the userspace DMU-OSS. For comparison, you can also see > the results of running the same PIOS benchmark with normal POSIX I/O on > the Solaris 10u3 ZFS implementation. > > See the attached PDF for the results. > > First page provides PIOS throughput information on striped pools with > 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, > the kernel ZFS implementation scales better to 24 and 46 disks. > Surprisingly enough, disabling checksums has a negative impact on the > throughput with a 10-disk striped pool. This impact can be attributed to > the differences in the uDMU IO pipeline which are probably causing the > IOs to not being parallelized when checksumming is disabled. This issue, > however, should not be hard to fix. > > Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and > mirrored pools. In these configurations, the kernel ZFS currently has a > better throughput than the userspace DMU. > > These results were obtained with only minimal tuning of the DMU (setting > a maximum cache size and increasing the number of I/O threads). Work is > already underway to improve the performance of the DMU and we expect to > see better throughput in the coming months. > > Best regards, > Ricardo > > -- > <http://www.sun.com> * Ricardo Manuel Correia * > > Lustre Engineering Group > *Sun Microsystems, Inc.* > Portugal > > Ricardo.M.Correia at Sun.COM >> _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss--
It would also be nice to see the tests repeated on Linux on the same hardware. (Obviously not the kernel ZFS stuff though!) Jim p.s. I should add that the efforts at Livermore to test the ZFS/PIOS stuff are on hold until we can get 1.6.2 stablized at scale for our production environment, likely another two months. Sorry we were not of more help. On Wed, 14 Nov 2007, Jody McIntyre wrote:> Hi Ricardo, > > On Tue, Nov 13, 2007 at 11:48:38AM +0000, Ricardo Correia wrote: > >> Here are some current ZFS/DMU benchmark results that might be of >> interest to you. > > It''s nice to see these results - thanks for posting them. When you have > time, could you do a comparison with Lustre 1.6.x on ldiskfs + software > RAID 5? It would be good to know how ZFS stands up (probably quite > favourably, even at this early stage.) > > Cheers, > Jody > >> These results were obtained on a Sunfire x4500 with these specifications: >> >> - 2 dual-core AMD Opteron processors >> - 16 GB main memory >> - 48 SATA II HDD (7200 RPM, 500 GB each) >> - Solaris 10 update 3 >> - Userspace DMU from OpenSolaris build 74 >> >> The tool used in this particular benchmark was PIOS, which simulates a >> parallel I/O load typically experienced by an OSS. Since PIOS links >> directly with the DMU, it should give a good estimation of the maximum >> throughput of the userspace DMU-OSS. For comparison, you can also see >> the results of running the same PIOS benchmark with normal POSIX I/O on >> the Solaris 10u3 ZFS implementation. >> >> See the attached PDF for the results. >> >> First page provides PIOS throughput information on striped pools with >> 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, >> the kernel ZFS implementation scales better to 24 and 46 disks. >> Surprisingly enough, disabling checksums has a negative impact on the >> throughput with a 10-disk striped pool. This impact can be attributed to >> the differences in the uDMU IO pipeline which are probably causing the >> IOs to not being parallelized when checksumming is disabled. This issue, >> however, should not be hard to fix. >> >> Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and >> mirrored pools. In these configurations, the kernel ZFS currently has a >> better throughput than the userspace DMU. >> >> These results were obtained with only minimal tuning of the DMU (setting >> a maximum cache size and increasing the number of I/O threads). Work is >> already underway to improve the performance of the DMU and we expect to >> see better throughput in the coming months. >> >> Best regards, >> Ricardo >> >> -- >> <http://www.sun.com> * Ricardo Manuel Correia * >> >> Lustre Engineering Group >> *Sun Microsystems, Inc.* >> Portugal >> >> Ricardo.M.Correia at Sun.COM >> > > >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > > -- > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel >
Hi, If you want to run the ZFS/DMU benchmarks yourself in Linux or Solaris, you will find instructions in the following wiki page: http://wiki.lustre.org/index.php?title=PIOS-DMU There are also instructions for running the Lustre uOSS alpha/demo on both operating systems here: http://wiki.lustre.org/index.php?title=Lustre_OSS/MDS_with_ZFS_DMU Let me know if you run into any problems. Best regards, Ricardo -- <http://www.sun.com> *Ricardo Manuel Correia* Lustre Engineering *Sun Microsystems, Inc.* Portugal -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071120/55348a34/attachment-0002.html -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20071120/55348a34/attachment-0002.gif
It appears, from what I?ve read, that the future of Lustre will be based on ZFS running in User space. What are the advantages to doing this in User space? I?ve also read that this will be available in Lustre 1.8. What is the timeframe for this? Thanks. -Roger _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/connect.html?ocid=TXT_TAGLM_Wave2_newways_112007 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.lustre.org/pipermail/lustre-devel/attachments/20071128/21f31cc5/attachment-0004.html