Team During a ZFS presentation I had a question from Vernon which I could not answer and did not find with a quick look through the archives. Q: What''s the effect (if any) of only having on Floating Point Processor on Niagara when you turn on ZFS compression? -- ============================================================= Trevor Pretty Mob: +64 21 666 161 Systems Engineer & OS Ambassador DDI: +64 9 976 6802 Sun Microsystems (NZ) Ltd. Fax: +64 9 976 6877 Level 5, 10 Viaduct Harbour Ave, PO Box 5766, Auckland, New Zealand ==============================================================
On 8/20/06, trevor pretty <Trevor.Pretty at sun.com> wrote:> Team > > During a ZFS presentation I had a question from Vernon which I could not > answer and did not find with a quick look through the archives. > > Q: What''s the effect (if any) of only having on Floating Point Processor > on Niagara when you turn on ZFS compression? >not an expert, but most if not all compression is integer based, and I don''t think floating point is supported inside the kernel anyway so it has to be integer based. James Dickens uadmin.blogspot.com> -- > =============================================================> Trevor Pretty Mob: +64 21 666 161 > Systems Engineer & OS Ambassador DDI: +64 9 976 6802 > Sun Microsystems (NZ) Ltd. Fax: +64 9 976 6877 > Level 5, 10 Viaduct Harbour Ave, > PO Box 5766, Auckland, New Zealand > =============================================================> _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On 8/20/06, James Dickens <jamesd.wi at gmail.com> wrote:> On 8/20/06, trevor pretty <Trevor.Pretty at sun.com> wrote: > > Team > > > > During a ZFS presentation I had a question from Vernon which I could not > > answer and did not find with a quick look through the archives. > > > > Q: What''s the effect (if any) of only having on Floating Point Processor > > on Niagara when you turn on ZFS compression? > > > not an expert, but most if not all compression is integer based, and > I don''t think floating point is supported inside the kernel anyway so > it has to be integer based.Not too long ago Roch said "compression runs in the context of a single thread per pool", which makes me worry much more about the speed of a single core doing all of the compression for a pool. Mike -- Mike Gerdts http://mgerdts.blogspot.com/
On 8/20/06, Mike Gerdts <mgerdts at gmail.com> wrote:> On 8/20/06, James Dickens <jamesd.wi at gmail.com> wrote: > > On 8/20/06, trevor pretty <Trevor.Pretty at sun.com> wrote: > > > Team > > > > > > During a ZFS presentation I had a question from Vernon which I could not > > > answer and did not find with a quick look through the archives. > > > > > > Q: What''s the effect (if any) of only having on Floating Point Processor > > > on Niagara when you turn on ZFS compression? > > > > > not an expert, but most if not all compression is integer based, and > > I don''t think floating point is supported inside the kernel anyway so > > it has to be integer based. > > Not too long ago Roch said "compression runs in the context of a > single thread per > pool", which makes me worry much more about the speed of a single core > doing all of the compression for a pool. >i hope that changes, at least per filesystem, or unlimited compression threads would be a great thing James uadmin.blogspot.com> Mike > > -- > Mike Gerdts > http://mgerdts.blogspot.com/ >
On Sun, Aug 20, 2006 at 06:16:23PM -0500, Mike Gerdts wrote:> On 8/20/06, James Dickens <jamesd.wi at gmail.com> wrote: > >On 8/20/06, trevor pretty <Trevor.Pretty at sun.com> wrote: > >> Team > >> > >> During a ZFS presentation I had a question from Vernon which I could not > >> answer and did not find with a quick look through the archives. > >> > >> Q: What''s the effect (if any) of only having on Floating Point Processor > >> on Niagara when you turn on ZFS compression? > >> > >not an expert, but most if not all compression is integer based, and > >I don''t think floating point is supported inside the kernel anyway so > >it has to be integer based.That''s correct, we don''t do any floating-point math in ZFS, either compression or checksumming. So Niagara''s floating point performance will have no effect on ZFS performance.> Not too long ago Roch said "compression runs in the context of a > single thread per pool", which makes me worry much more about the > speed of a single core doing all of the compression for a pool.This was not the design, we''re working on fixing this bug so that many threads will be used to do the compression. --matt
Matthew, On 8/20/06 6:20 PM, "Matthew Ahrens" <ahrens at eng.sun.com> wrote:> This was not the design, we''re working on fixing this bug so that many > threads will be used to do the compression.Is this also true of decompression? - Luke
On our Niagara T2000 32x1000mhz box, 8 gigs ram, 4x68gig disk drives . we setup three drives as raidz with compression. all of our performance issues are gone. remember we receive ~150 million lines of ASCII and 2 million files per day . we have had zero performance issues since we 1) upgraded so solaris u03 2) added two more drives and setup a 3 disk raidz while doing this we also turned on compression for the heck of it. no performance differences we can measure. we are satisfied now with both the Niagara and ZFS. the web site is faster also http://canary.sfbay. try looking at a report where we must processs 100,000 lines of data. http://itsm-mpk-2.sfbay/canary/cgi-bin/canary.cgi?group=report&rpt_source=local&local_rpt=process_count&hour=11&r1=3&r2=1 This report shows every process running on every sun ray server in the world (approx 1 million pids). you can scroll down this report and find out which of the 1500 unique executables (oracle, vi,vim,emacs,mozilla,firefox,thunderbird,opera,.etc.) is causing the most load across the world. basically our performance issues have been solved. thanks ZFS and Niagara teams. sean Matthew Ahrens wrote:>On Sun, Aug 20, 2006 at 06:16:23PM -0500, Mike Gerdts wrote: > > >>On 8/20/06, James Dickens <jamesd.wi at gmail.com> wrote: >> >> >>>On 8/20/06, trevor pretty <Trevor.Pretty at sun.com> wrote: >>> >>> >>>>Team >>>> >>>>During a ZFS presentation I had a question from Vernon which I could not >>>>answer and did not find with a quick look through the archives. >>>> >>>>Q: What''s the effect (if any) of only having on Floating Point Processor >>>>on Niagara when you turn on ZFS compression? >>>> >>>> >>>> >>>not an expert, but most if not all compression is integer based, and >>>I don''t think floating point is supported inside the kernel anyway so >>>it has to be integer based. >>> >>> > >That''s correct, we don''t do any floating-point math in ZFS, either >compression or checksumming. So Niagara''s floating point performance >will have no effect on ZFS performance. > > > >>Not too long ago Roch said "compression runs in the context of a >>single thread per pool", which makes me worry much more about the >>speed of a single core doing all of the compression for a pool. >> >> > >This was not the design, we''re working on fixing this bug so that many >threads will be used to do the compression. > >--matt >_______________________________________________ >zfs-discuss mailing list >zfs-discuss at opensolaris.org >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >-- <http://www.sun.com> * Sean Meighan * Mgr ITSM Engineering *Sun Microsystems, Inc.* US Phone x32329 / +1 408 850-9537 Mobile 303-520-2024 Fax 408 850-9537 Email Sean.Meighan at Sun.COM ------------------------------------------------------------------------ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060820/3ae1a967/attachment.html>
On Sun, 20 Aug 2006, Sean Meighan wrote:> On our Niagara T2000 32x1000mhz box, 8 gigs ram, 4x68gig disk drives . we > setup three drives as raidz with compression. all of our performance issues > are gone. remember we receive ~150 million lines of ASCII and 2 million files > per day . we have had zero performance issues since we > 1) upgraded so solaris u03 > 2) added two more drives and setup a 3 disk raidzWow, those are impressive stats! For the sake of us outside the SWAN, can you please give us a brief idea of what this machine (canary.sfbay?) does? Cheers, -- Rich Teer, SCNA, SCSA, OpenSolaris CAB member President, Rite Online Inc. Voice: +1 (250) 979-1638 URL: http://www.rite-group.com/rich
On Sun, Aug 20, 2006 at 08:38:03PM -0700, Luke Lonergan wrote:> Matthew, > > On 8/20/06 6:20 PM, "Matthew Ahrens" <ahrens at eng.sun.com> wrote: > > > This was not the design, we''re working on fixing this bug so that many > > threads will be used to do the compression. > > Is this also true of decompression?I believe that decompression already runs in many threads. If you see differently, let us know. --matt
canary is a set of scripts that runs on all sun ray servers. We have a Solaris, Linux and Windows version of these scripts. I have attached the solaris script. I have attached the output file (it has embedded HTML so view it in a browser). These scripts produce about 1000 lines of ASCII that are forwarded to a central server At Sun we have 752 servers (30,000 employees) that send these files to a single Niagara T2000 box (itsm-mpk-2.sfbay) running ZFS. This single server shows the activity for the world. We wrote the central server in Perl and it is heavily multi-threaded. Here is the cpu load on our central server for the last 2 weeks (Remember 150 million lines of data per day). On our graphs we have Y1 and Y2 axes. Look at the arrow in front of the legend to know which axis to use. We have 8 gigs of memory but we never run out. We think a single CPU laptop running as a canary server could service 50 clients. We are working to make the canary software free through open source, this is currently in progress. As a side note we are at version 163 of the canary client, we use an expect script to push this code out to the client machines. Since the canary never needs root on either the client or server we can do this easily. We can install a new client version on the 750 servers in under three minutes with a single command. This means we have installed over installed the canary client over 100,000 times over the last two years and have never affected production. We had some early problems with some programs (ping,traceroute,df) that would hag. We now put about 10 of these commands into the background and if they are not finished in 90 seconds we terminate them. This solved the hanging issue. thanks sean Rich Teer wrote:>On Sun, 20 Aug 2006, Sean Meighan wrote: > > > >>On our Niagara T2000 32x1000mhz box, 8 gigs ram, 4x68gig disk drives . we >>setup three drives as raidz with compression. all of our performance issues >>are gone. remember we receive ~150 million lines of ASCII and 2 million files >>per day . we have had zero performance issues since we >>1) upgraded so solaris u03 >>2) added two more drives and setup a 3 disk raidz >> >> > >Wow, those are impressive stats! For the sake of us outside the SWAN, can >you please give us a brief idea of what this machine (canary.sfbay?) does? > >Cheers, > > >-- <http://www.sun.com> * Sean Meighan * Mgr ITSM Engineering *Sun Microsystems, Inc.* US Phone x32329 / +1 408 850-9537 Mobile 303-520-2024 Fax 408 850-9537 Email Sean.Meighan at Sun.COM ------------------------------------------------------------------------ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. ------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/63e9119b/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: itsm-mpk-2.SFBay.Sun.com_cpu.png Type: image/png Size: 14197 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/63e9119b/attachment.png> -------------- next part -------------- A non-text attachment was scrubbed... Name: itsm-mpk-2.SFBay.Sun.com_mem.png Type: image/png Size: 11750 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/63e9119b/attachment-0001.png> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: canary_probe_solaris.sh URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/63e9119b/attachment.ksh> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: canary.rawdatafile.canary URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/63e9119b/attachment-0001.ksh>
Mike Gerdts wrote:>> not an expert, but most if not all compression is integer based, and >> I don''t think floating point is supported inside the kernel anyway so >> it has to be integer based. > > Not too long ago Roch said "compression runs in the context of a > single thread per > pool", which makes me worry much more about the speed of a single core > doing all of the compression for a pool.I haven''t done measurements of this in years, but... I''ll wager that compression is memory bound, not CPU bound, for today''s servers. A system with low latency and high bandwidth memory will perform well (UltraSPARC-T1). Threading may not help much on systems with a single memory interface, but should help some on systems with multiple memory interfaces (UltraSPARC-*, Opteron, Athlon FX, etc.) -- richard
On 8/21/06, Richard Elling - PAE <Richard.Elling at sun.com> wrote:> I haven''t done measurements of this in years, but... > I''ll wager that compression is memory bound, not CPU bound, for today''s > servers. A system with low latency and high bandwidth memory will perform > well (UltraSPARC-T1). Threading may not help much on systems with a > single > memory interface, but should help some on systems with multiple memory > interfaces (UltraSPARC-*, Opteron, Athlon FX, etc.) > -- richard >A rather simple test using CSQamp.pkg from the cooltools download site. There''s nothing magical about this file - it just happens to be a largish file that I had on hand. $ time gzip -c < CSQamp.pkg > /dev/null V40z: real 0m15.339s user 0m14.534s sys 0m0.485s V240: real 0m35.825s user 0m35.335s sys 0m0.284s T2000: time gzip -c < CSQamp.pkg > /dev/null real 1m33.669s user 1m32.768s sys 0m0.881s If I do 8 gzips in parallel: V40z: time ~/scripts/pgzip real 0m32.632s user 1m53.382s sys 0m1.653s V240: time ~/scripts/pgzip real 2m24.704s user 4m42.430s sys 0m2.305s T2000: time ~/scripts/pgzip real 1m40.165s user 13m10.475s sys 0m6.578s In each of the tests, the file was in /tmp. As expected, the V40z running 8 gzip processes (using 4 cores) took twice as long as it did running 1 (using 1 core). The V240 took 4 times as long (8 processes, 2 threads) as the original, and the T2000 ran 8 (8 processes, 8 cores) in just about the same amount of time as it ran 1. For giggles, I ran 32 processes on the T2000 and came up with 5m4.585s(real) 158m33.380s (user) and 42.484s (sys). In other words, the T2000 running 32 gzip processes had an elapsed time of 3 times greater than 8 processes. Even though the elapsed jumped by 3x, the %sys jumped by nearly 7x. Here''s a summary: Server gzips Seconds MB/sec V40z 8 32.632 49,445 T2000 32 304.585 21,189 T2000 8 100.165 16,108 V40z 1 15.339 13,149 V240 8 144.704 11,150 V240 1 35.825 5,630 T2000 1 99.669 2,024 Clearly more threads doing compression with gzip give better performance than a single thread. How that translates into memory vs. CPU speed, I am not sure. However, I can''t help but think that if my file server is compressing every data block that it writes that it would be able to write more data if it used a thread (or more) per core I would come out ahead. I am a firm believer that the next generation of compression commands and libraries need to use parallel algorithms. The simplest way to do this would be to divide the data into chunks and farm out each chunk to various worker threads. This will likely come at the cost of efficiency of the compression, but in intial tests I have done this amounts to a very small difference in size relative to the speedup achieved. Initial tests were with a chunk of C code and zlib. Mike -- Mike Gerdts http://mgerdts.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060821/cca130cd/attachment.html>
However, I can''t help but think that if my file server is compressing every data block that it writes that it would be able to write more data if it used a thread (or more) per core I would come out ahead. No arguments here. MT-hot compression was the design of ZFS from day one. A bug got in the way and made compression singled threaded per zpool. -r ____________________________________________________________________________________ Performance, Availability & Architecture Engineering Roch Bourbonnais Sun Microsystems, Icnc-Grenoble Senior Performance Analyst 180, Avenue De L''Europe, 38330, Montbonnot Saint Martin, France http://icncweb.france/~rbourbon http://blogs.sun.com/roller/page/roch Roch.Bourbonnais at Sun.Com (+33).4.76.18.83.20