With the CPU overhead imposed in checksum of blocks by ZFS, on a large sequential write test, the CPU was heavily loaded in a test that I ran. By turning off the checksum, the CPU load was greatly reduced. Obviously, this caused a tradeoff in reliability for CPU cycles. Would the logic behind ZFS take full advantage of a heavily multicored system, such as on the Sun Niagara platform? Would it utilize of the 32 concurrent threads for generating its checksums? Has anyone compared ZFS on a Sun Tx000, to that of a 2-4 thread x64 machine? This message posted from opensolaris.org
>With the CPU overhead imposed in checksum of blocks by ZFS, on a large >sequential write test, the CPU was heavily loaded in a test that I ran. >By turning off the checksum, the CPU load was greatly reduced. >Obviously, this caused a tradeoff in reliability for CPU cycles.What hardware platform and what was the I/O throughput at the peak and what was the difference in throughput and CPU utilization between both cases? Casper
> Would the logic behind ZFS take full advantage of a heavily multicored > system, such as on the Sun Niagara platform? Would it utilize of the > 32 concurrent threads for generating its checksums? Has anyone > compared ZFS on a Sun Tx000, to that of a 2-4 thread x64 machine?Pete and I are working on resolving ZFS scalability issues with Niagara and StarCat right now. I''m not sure if any official numbers about ZFS performance on Niagara have been published. As far as concurrent threads generating checksums goes, the system doesn''t work quite the way you have postulated. The checksum is generated in the ZIO_STAGE_CHECKSUM_GENERATE pipeline state for writes, and verified in the ZIO_STAGE_CHECKSUM_VERIFY pipeline stage for reads. Whichever thread happens to advance the pipline to the checksum generate stage is the thread that will actually perform the work. ZFS does not break the work of the checksum into chunks and have multiple CPUs perform the computation. However, it is possible to have concurrent writes simultaneously in the checksum_generate stage. More details about this can be found in zfs/zio.c and zfs/sys/zio_impl.h -j
The experiment was on a V240. Throughput wasn''t the issue in our test; CPU utilization seemed to drop by approx. 50% after turning checksum off. The concern was in potentially running out of CPU horsepower to support multiple parallel sequential writes. This message posted from opensolaris.org
Thanks for that info. I validated with a simple experiment on a Niagara machine, by viewing ''mpstat'' that no more than 2-3 threads were being saturated by my large block sequential write test. This message posted from opensolaris.org
I''ve seen very good performance on streaming large files to ZFS on a T2000. We have been looking at using the T2000 as a disk storage unit for backups. I''ve been able to push over 500MB/s to the disks. Setup is EMC Clariion CX3 with 84 500GB SATA drives connected w/ 4Gbps all the way to the disk shelves. The 84 drives are presented as raw luns to the T2000 -- no HW RAID enabled on the Clariion. The problem we''ve seen comes when enabling compression, as that is single threaded per zpool. Enabling compression drops our throughput to 12-15MB/s per pool. This is bugid: 6460622, the fix is apparently set to be put back into Nevada fairly soon. -Andy Reuven Kaswin wrote:> With the CPU overhead imposed in checksum of blocks by ZFS, on a large sequential write test, the CPU was heavily loaded in a test that I ran. By turning off the checksum, the CPU load was greatly reduced. Obviously, this caused a tradeoff in reliability for CPU cycles. > > Would the logic behind ZFS take full advantage of a heavily multicored system, such as on the Sun Niagara platform? Would it utilize of the 32 concurrent threads for generating its checksums? Has anyone compared ZFS on a Sun Tx000, to that of a 2-4 thread x64 machine? > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >
On 09 February, 2007 - Reuven Kaswin sent me these 0,4K bytes:> Thanks for that info. I validated with a simple experiment on a > Niagara machine, by viewing ''mpstat'' that no more than 2-3 threads > were being saturated by my large block sequential write test.And on, say 32 parallel writes? /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
On Feb 9, 2007, at 8:02 AM, Carisdad wrote:> I''ve seen very good performance on streaming large files to ZFS on > a T2000. We have been looking at using the T2000 as a disk storage > unit for backups. I''ve been able to push over 500MB/s to the > disks. Setup is EMC Clariion CX3 with 84 500GB SATA drives > connected w/ 4Gbps all the way to the disk shelves. The 84 drives > are presented as raw luns to the T2000 -- no HW RAID enabled on the > Clariion. The problem we''ve seen comes when enabling compression, > as that is single threaded per zpool. Enabling compression drops > our throughput to 12-15MB/s per pool. > This is bugid: 6460622, the fix is apparently set to be put back > into Nevada fairly soon.This was just putback and will be in snv_59. enjoy your unconstrained compressed I/O, eric
eric kustarz wrote:> > On Feb 9, 2007, at 8:02 AM, Carisdad wrote: > >> I''ve seen very good performance on streaming large files to ZFS on a >> T2000. We have been looking at using the T2000 as a disk storage >> unit for backups. I''ve been able to push over 500MB/s to the disks. >> Setup is EMC Clariion CX3 with 84 500GB SATA drives connected w/ >> 4Gbps all the way to the disk shelves. The 84 drives are presented >> as raw luns to the T2000 -- no HW RAID enabled on the Clariion. The >> problem we''ve seen comes when enabling compression, as that is single >> threaded per zpool. Enabling compression drops our throughput to >> 12-15MB/s per pool. >> This is bugid: 6460622, the fix is apparently set to be put back into >> Nevada fairly soon. > > This was just putback and will be in snv_59. > > enjoy your unconstrained compressed I/O, > eric > >Awesome, thank you. I almost hate to ask, but... Is there any way to track when the fix will make its way back into Solaris 10 via update4(?) or a patch? Thanks again, -Andy
On Feb 22, 2007, at 10:01 AM, Carisdad wrote:> eric kustarz wrote: >> >> On Feb 9, 2007, at 8:02 AM, Carisdad wrote: >> >>> I''ve seen very good performance on streaming large files to ZFS >>> on a T2000. We have been looking at using the T2000 as a disk >>> storage unit for backups. I''ve been able to push over 500MB/s to >>> the disks. Setup is EMC Clariion CX3 with 84 500GB SATA drives >>> connected w/ 4Gbps all the way to the disk shelves. The 84 >>> drives are presented as raw luns to the T2000 -- no HW RAID >>> enabled on the Clariion. The problem we''ve seen comes when >>> enabling compression, as that is single threaded per zpool. >>> Enabling compression drops our throughput to 12-15MB/s per pool. >>> This is bugid: 6460622, the fix is apparently set to be put back >>> into Nevada fairly soon. >> >> This was just putback and will be in snv_59. >> >> enjoy your unconstrained compressed I/O, >> eric >> >> > Awesome, thank you. > > I almost hate to ask, but... Is there any way to track when the > fix will make its way back into Solaris 10 via update4(?) or a patch?we''re gonna try and squeeze it into s10u4, no guarantees just yet though. If it makes it, i''ll update the list... eric
Hello eric, Thursday, February 22, 2007, 5:44:35 PM, you wrote: ek> On Feb 9, 2007, at 8:02 AM, Carisdad wrote:>> I''ve seen very good performance on streaming large files to ZFS on >> a T2000. We have been looking at using the T2000 as a disk storage >> unit for backups. I''ve been able to push over 500MB/s to the >> disks. Setup is EMC Clariion CX3 with 84 500GB SATA drives >> connected w/ 4Gbps all the way to the disk shelves. The 84 drives >> are presented as raw luns to the T2000 -- no HW RAID enabled on the >> Clariion. The problem we''ve seen comes when enabling compression, >> as that is single threaded per zpool. Enabling compression drops >> our throughput to 12-15MB/s per pool. >> This is bugid: 6460622, the fix is apparently set to be put back >> into Nevada fairly soon.ek> This was just putback and will be in snv_59. ek> enjoy your unconstrained compressed I/O, That is great news! Now I have to ask thru offician channels for a backport to s10. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com