zfs discuss - Dec 2010 - stupid ZFS question - floating point operations

I have a coworker, who''s primary expertise is in another flavor of
Unix.

This coworker lists floating point operations as one of ZFS detriments.

I''s not really sure what he means specifically, or where he got this
reference from.

In an effort to refute what I believe is an error or misunderstanding on
his part, I have spent time on Yahoo, Google, the ZFS section of
OpenSolaris.org, etc.  I really haven''t turned up much of anything that
would prove or disprove his comments.  The one thing I haven''t done is
to go through the ZFS source code, but its been years since I have done
any serious programming.

If someone from Oracle, or anyone on this mailing list could point me
towards any documentation, or give me a definitive word, I would sure
appreciate it.  If there were floating point operations going on within
ZFS, at this point I am uncertain as to what they would be.

TIA for any comments,

Jerry

On 22 December, 2010 - Jerry Kemp sent me these 1,0K bytes:
> I have a coworker, who''s primary expertise is in another flavor of
Unix.
> 
> This coworker lists floating point operations as one of ZFS detriments.
> 
> I''s not really sure what he means specifically, or where he got
this
> reference from.
Then maybe ask him first? Guilty until proven innocent isn''t the
regular
path...
> In an effort to refute what I believe is an error or misunderstanding on
> his part, I have spent time on Yahoo, Google, the ZFS section of
> OpenSolaris.org, etc.  I really haven''t turned up much of anything
that
> would prove or disprove his comments.  The one thing I haven''t
done is
> to go through the ZFS source code, but its been years since I have done
> any serious programming.
> 
> If someone from Oracle, or anyone on this mailing list could point me
> towards any documentation, or give me a definitive word, I would sure
> appreciate it.  If there were floating point operations going on within
> ZFS, at this point I am uncertain as to what they would be.
> 
> TIA for any comments,
> 
> Jerry
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se

On 12/23/10 08:44 AM, Jerry Kemp wrote:
> I have a coworker, who''s primary expertise is in another flavor of
Unix.
>
> This coworker lists floating point operations as one of ZFS detriments.
>
> I''s not really sure what he means specifically, or where he got
this
> reference from.
>
It sounds like your colleague is from another planet where filesystems 
perform computations!

-- 
Ian.

On 22/12/10 2:44 PM, Jerry Kemp wrote:
> I have a coworker, who''s primary expertise is in another flavor of
Unix.
> 
> This coworker lists floating point operations as one of ZFS detriments.
> 
Perhaps he can point you also to the equally mythical competing
filesystem which offers ZFS'' advantages.

--Toby
> I''s not really sure what he means specifically, or where he got
this
> reference from.
> ...
> Jerry
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

If I remember correctly Solaris like most other operating system does not save
or restore the floating point registers when context switching from User to
Kernel so doing any floating point ops in the kernel would corrupt user floating
point state. This means ZFS cannot be doing any floating point ops in the kernel
context.

Others wiser than I may be able to asert this with more certainty 

-Angelo


On Dec 22, 2010, at 2:44 PM, Jerry Kemp wrote:
> I have a coworker, who''s primary expertise is in another flavor of
Unix.
> 
> This coworker lists floating point operations as one of ZFS detriments.
> 
> I''s not really sure what he means specifically, or where he got
this
> reference from.
> 
> In an effort to refute what I believe is an error or misunderstanding on
> his part, I have spent time on Yahoo, Google, the ZFS section of
> OpenSolaris.org, etc.  I really haven''t turned up much of anything
that
> would prove or disprove his comments.  The one thing I haven''t
done is
> to go through the ZFS source code, but its been years since I have done
> any serious programming.
> 
> If someone from Oracle, or anyone on this mailing list could point me
> towards any documentation, or give me a definitive word, I would sure
> appreciate it.  If there were floating point operations going on within
> ZFS, at this point I am uncertain as to what they would be.
> 
> TIA for any comments,
> 
> Jerry
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On 12/22/2010 11:49 AM, Tomas ?gren wrote:
> On 22 December, 2010 - Jerry Kemp sent me these 1,0K bytes:
>
>> I have a coworker, who''s primary expertise is in another
flavor of Unix.
>>
>> This coworker lists floating point operations as one of ZFS detriments.
>>
>> I''s not really sure what he means specifically, or where he
got this
>> reference from.
> Then maybe ask him first? Guilty until proven innocent isn''t the
regular
> path...
>
>> In an effort to refute what I believe is an error or misunderstanding
on
>> his part, I have spent time on Yahoo, Google, the ZFS section of
>> OpenSolaris.org, etc.  I really haven''t turned up much of
anything that
>> would prove or disprove his comments.  The one thing I haven''t
done is
>> to go through the ZFS source code, but its been years since I have done
>> any serious programming.
>>
>> If someone from Oracle, or anyone on this mailing list could point me
>> towards any documentation, or give me a definitive word, I would sure
>> appreciate it.  If there were floating point operations going on within
>> ZFS, at this point I am uncertain as to what they would be.
>>
>> TIA for any comments,
>>
>> Jerry
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
> /Tomas
So far as my understanding of the codebase goes (and, while I''ve read a
significant portion, I''m not really an expert here):

Assuming he means that ZFS has a weakness of heavy floating-point 
calculation requirements (i.e using ZFS requires heavy FP usage),
that''s
wrong.

Like all normal filesystems, the "ordinary" operations are all
integer,
load, and store.  The ordinary work of caching, block allocation, and 
fetching/writing is of course all integer-based. I can''t imagine
someone
writing a filesystem which does such operations using floating point.

A quick grep through the main ZFS sources doesn''t find anything of type
"double" or "float".

I think he might be confused with what is happening on Checksums (which 
is still all Integer, but looks/sounds "expensive").  Yes, ZFS is 
considerably *more* compute intensive than other filesystems.  However, 
it''s all Integer, and one of the base assumptions of ZFS is that modern
systems have lots of excess CPU cycles around, so stealing 5% for use 
with ZFS won''t impact performance much, and the added features of ZFS 
more than make up for any CPU cycles lost.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

Generally, ZFS does not use floating point.

And further, use of floating point in the kernel is exceptionally rare.  The
kernel does not save floating point context automatically, which means that code
that uses floating point needs to take special care to make sure any context
from userland is saved and restored before it can use the registers itself.  
This rather onerous burden tends to exclude "easy" consumption of
floating point.

That said, some operations -- and cryptographic ones in particular -- may use
floating point registers and operations because for some architectures (sun4u
rings a bell) this can make certain expensive operations go faster.  I
don''t think this is the case for secure hash/message digest algorithms,
but if you use ZFS encryption as found in Solaris 11 Express you might find that
on certain systems these registers are used for performance reasons, either on
the bulk crypto or on the keying operations.  (More likely the latter, but my
memory of these optimizations is still hazy.)

Note that *if* this is done, it is only done where such an operation is a
performance win, and not because any of the math is inherently floating point. 
So in this case, I would say that this optimization would be an advantage,
rather than a disadvantage.

Oh, and this usage only applies to Solaris, and is optional.  I doubt FreeBSD
has these particular enhancements -- indeed, IIRC, these optimizations are
specific to certain classes of SPARC cpus and probably are not performed at all
for x86 cpus.

  - Garrett


-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org on behalf of Erik Trimble
Sent: Wed 12/22/2010 12:08 PM
To: Jerry Kemp; zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] stupid ZFS question - floating point operations
 
On 12/22/2010 11:49 AM, Tomas ?gren wrote:
> On 22 December, 2010 - Jerry Kemp sent me these 1,0K bytes:
>
>> I have a coworker, who''s primary expertise is in another
flavor of Unix.
>>
>> This coworker lists floating point operations as one of ZFS detriments.
>>
>> I''s not really sure what he means specifically, or where he
got this
>> reference from.
> Then maybe ask him first? Guilty until proven innocent isn''t the
regular
> path...
>
>> In an effort to refute what I believe is an error or misunderstanding
on
>> his part, I have spent time on Yahoo, Google, the ZFS section of
>> OpenSolaris.org, etc.  I really haven''t turned up much of
anything that
>> would prove or disprove his comments.  The one thing I haven''t
done is
>> to go through the ZFS source code, but its been years since I have done
>> any serious programming.
>>
>> If someone from Oracle, or anyone on this mailing list could point me
>> towards any documentation, or give me a definitive word, I would sure
>> appreciate it.  If there were floating point operations going on within
>> ZFS, at this point I am uncertain as to what they would be.
>>
>> TIA for any comments,
>>
>> Jerry
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
> /Tomas
So far as my understanding of the codebase goes (and, while I''ve read a
significant portion, I''m not really an expert here):

Assuming he means that ZFS has a weakness of heavy floating-point 
calculation requirements (i.e using ZFS requires heavy FP usage),
that''s
wrong.

Like all normal filesystems, the "ordinary" operations are all
integer,
load, and store.  The ordinary work of caching, block allocation, and 
fetching/writing is of course all integer-based. I can''t imagine
someone
writing a filesystem which does such operations using floating point.

A quick grep through the main ZFS sources doesn''t find anything of type
"double" or "float".

I think he might be confused with what is happening on Checksums (which 
is still all Integer, but looks/sounds "expensive").  Yes, ZFS is 
considerably *more* compute intensive than other filesystems.  However, 
it''s all Integer, and one of the base assumptions of ZFS is that modern
systems have lots of excess CPU cycles around, so stealing 5% for use 
with ZFS won''t impact performance much, and the added features of ZFS 
more than make up for any CPU cycles lost.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20101222/745c630d/attachment.html>

Thank you to everyone who replied to my question.

Apparently, the place that I had not looked were the ZFS list archives
themselves.  Someone else had already asked the question, and it was
answered by Matthew Ahrens @ Sun Microsystems in this thread here.

http://mail.opensolaris.org/pipermail/zfs-discuss/2006-August/004574.html

To provide some more background, the coworker in question had the
understanding that ZFS was using floating point to do checksums, and had
come to this understanding from a Solaris 9->10 migration presentation
that had occurred several years earlier.

again, thank you to everyone who commented.

Jerry

There are no floating points operations in zfs, however even if there would
that wouldn''t be a bad thing, as modern CPU are float monsters indeed
its
likely some things would be faster if converted to use the float ALU (note
however those operations would have to account for the different properties
of the ALUs).

The most complex ALU parts of zfs are the Galois field math in the RAID
codes and the checksum operations. All of which are integer ops (tho it
might be interesting to see if the Galois field could be accelerated using
the float ALU).

Deano

-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Jerry Kemp
Sent: 22 December 2010 19:44
To: zfs-discuss at opensolaris.org
Subject: [zfs-discuss] stupid ZFS question - floating point operations

I have a coworker, who''s primary expertise is in another flavor of
Unix.

This coworker lists floating point operations as one of ZFS detriments.

I''s not really sure what he means specifically, or where he got this
reference from.

In an effort to refute what I believe is an error or misunderstanding on
his part, I have spent time on Yahoo, Google, the ZFS section of
OpenSolaris.org, etc.  I really haven''t turned up much of anything that
would prove or disprove his comments.  The one thing I haven''t done is
to go through the ZFS source code, but its been years since I have done
any serious programming.

If someone from Oracle, or anyone on this mailing list could point me
towards any documentation, or give me a definitive word, I would sure
appreciate it.  If there were floating point operations going on within
ZFS, at this point I am uncertain as to what they would be.

TIA for any comments,

Jerry
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On 2010-Dec-23 04:48:19 +0800, Deano <deano at rattie.demon.co.uk>
wrote:
> modern CPU are float monsters indeed its
>likely some things would be faster if converted to use the float ALU
_Some_ modern CPUs are good at FP, a lot aren''t.  The SPARC T-1 was
particularly poor as it only had a single FPU.  Likewise, performance
in the x86 world is highly variable, depending on the vendor and core
you pick.  AFAIK, iA64 and PPC are consistently good - but neither are
commonly found in conjunction with ZFS.  You may also need to allow
for software assist:  Very few CPUs implement all of the IEEE FP
standard in hardware and most (including SPARC) require software to
implement parts of the standard.  If your algorithm happens to make
significant use of things other than normalised numbers and zero, your
performance may be severely affected by the resultant traps and
software assistance.

Any use of floating point within the kernel also means changes to
when FPU context is saved - and, unless this can be implemented
lazily, it will adversely impact the cost of all context switches
and potentially system calls.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20101223/6e2e1c78/attachment-0001.bin>

-----Original Message-----
From: Peter Jeremy [mailto:peter.jeremy at alcatel-lucent.com] 
Sent: 22 December 2010 21:17
To: Deano
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] stupid ZFS question - floating point operations

On 2010-Dec-23 04:48:19 +0800, Deano <deano at rattie.demon.co.uk>
wrote:
> modern CPU are float monsters indeed its likely some things would be 
>faster if converted to use the float ALU
Peter wrote
> _Some_ modern CPUs are good at FP, a lot aren''t.  The SPARC T-1
was
particularly
> poor as it only had a single FPU.  Likewise, performance in the x86 world
is highly
> variable, depending on the vendor and core you pick.  AFAIK, iA64 and PPC
are
> consistently good - but neither are commonly found in conjunction with
ZFS.  You
> may also need to allow for software assist:  Very few CPUs implement all
of the
> IEEE FP standard in hardware and most (including SPARC) require software
to
> implement parts of the standard.  If your algorithm happens to make
significant use
> of things other than normalised numbers and zero, your performance may be
severely
> affected by the resultant traps and software assistance.
I can''t speak for old architecture like SPARC but all modern ALU
designs
support most of the subset of useful IEEE in hardware and at high speed.
In particular x86 has extremely good float ALU performance, compared to some
architectures it is relatively low but certainly not something to avoid.

A CPU design that isn''t at *least* 5 GFLOPS per core is archaic. This
is
only accelerating due to the consumer market that many CPUs end up in. From
graphics to video decoding to audio synthesis, floating point math
dominants.

Not to say they aren''t able to perform very many integer ALU ops as
well,
just that the old mantra that FPU is to be avoided hasn''t been true for
years.
> Any use of floating point within the kernel also means changes to when FPU
context
> is saved - and, unless this can be implemented lazily, it will adversely
impact the
> cost of all context switches and potentially system calls.
Of course the cost of the extra register movement involved in context
switches is a concern, but this cost can be evaluated against the gains. I
like to see someone actually profile the costs in SunOs as many kernel
architectures I know accept FPU (and other specialist registers) restoration
when needed as a worthwhile cost.

Bye,
Deano

On 22/12/2010 20:27, Garrett D''Amore wrote:
> That said, some operations -- and cryptographic ones in particular --
> may use floating point registers and operations because for some
> architectures (sun4u rings a bell) this can make certain expensive
Well remembered!  There are sun4u optimisations that use the floating 
point unit but those only apply to the bignum code which in kernel is 
only used by RSA.
> operations go faster. I don''t think this is the case for secure
> hash/message digest algorithms, but if you use ZFS encryption as found
> in Solaris 11 Express you might find that on certain systems these
> registers are used for performance reasons, either on the bulk crypto or
> on the keying operations. (More likely the latter, but my memory of
> these optimizations is still hazy.)
RSA isn''t used at all by ZFS encryption, everything is AES (including 
key wrapping) and SHA256.

So those optimistations for floating point don''t come into play for ZFS
encryption.

-- 
Darren J Moffat

Thanks for the clarification.  I guess I need to go back and figure out how ZFS
crypto keying is performed.  I guess most likely the key is generated from some
sort of one-way hash from a passphrase?

  - Garrett

-----Original Message-----
From: Darren J Moffat [mailto:Darren.Moffat at oracle.com]
Sent: Thu 12/23/2010 1:32 AM
To: Garrett D''Amore
Cc: Erik Trimble; Jerry Kemp; zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] stupid ZFS question - floating point operations
 
On 22/12/2010 20:27, Garrett D''Amore wrote:
> That said, some operations -- and cryptographic ones in particular --
> may use floating point registers and operations because for some
> architectures (sun4u rings a bell) this can make certain expensive
Well remembered!  There are sun4u optimisations that use the floating 
point unit but those only apply to the bignum code which in kernel is 
only used by RSA.
> operations go faster. I don''t think this is the case for secure
> hash/message digest algorithms, but if you use ZFS encryption as found
> in Solaris 11 Express you might find that on certain systems these
> registers are used for performance reasons, either on the bulk crypto or
> on the keying operations. (More likely the latter, but my memory of
> these optimizations is still hazy.)
RSA isn''t used at all by ZFS encryption, everything is AES (including 
key wrapping) and SHA256.

So those optimistations for floating point don''t come into play for ZFS
encryption.

-- 
Darren J Moffat

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20101223/41bb977d/attachment.html>

Darren J Moffat <Darren.Moffat at oracle.com> wrote:
> On 22/12/2010 20:27, Garrett D''Amore wrote:
> > That said, some operations -- and cryptographic ones in particular --
> > may use floating point registers and operations because for some
> > architectures (sun4u rings a bell) this can make certain expensive
>
> Well remembered!  There are sun4u optimisations that use the floating 
> point unit but those only apply to the bignum code which in kernel is 
> only used by RSA.
It may be a guess caused by the fact that integer division and multiplication
is inside the FPU on SPARC processors.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Thu, Dec 23, 2010 at 09:32:13AM +0000, Darren J Moffat
wrote:
> On 22/12/2010 20:27, Garrett D''Amore wrote:
> >That said, some operations -- and cryptographic ones in particular --
> >may use floating point registers and operations because for some
> >architectures (sun4u rings a bell) this can make certain expensive
> 
> Well remembered!  There are sun4u optimisations that use the
> floating point unit but those only apply to the bignum code which in
> kernel is only used by RSA.
> 
> >operations go faster. I don''t think this is the case for
secure
> >hash/message digest algorithms, but if you use ZFS encryption as found
> >in Solaris 11 Express you might find that on certain systems these
> >registers are used for performance reasons, either on the bulk crypto
or
> >on the keying operations. (More likely the latter, but my memory of
> >these optimizations is still hazy.)
> 
> RSA isn''t used at all by ZFS encryption, everything is AES
> (including key wrapping) and SHA256.
> 
> So those optimistations for floating point don''t come into play
for
> ZFS encryption.
Moreover, we have platform-specific crypto optimizations.  If there were
FPU operations that help speed up symmetric crypto on an M4000 but not
on UltraSPARC T2s, then we''d use that on the one but not on the other.

Nico
--

On 23/12/2010 17:09, Joerg.Schilling at fokus.fraunhofer.de
wrote:
> Darren J Moffat<Darren.Moffat at oracle.com>  wrote:
>
>> On 22/12/2010 20:27, Garrett D''Amore wrote:
>>> That said, some operations -- and cryptographic ones in particular
--
>>> may use floating point registers and operations because for some
>>> architectures (sun4u rings a bell) this can make certain expensive
>>
>> Well remembered!  There are sun4u optimisations that use the floating
>> point unit but those only apply to the bignum code which in kernel is
>> only used by RSA.
>
> It may be a guess caused by the fact that integer division and
multiplication
> is inside the FPU on SPARC processors.
Not a guess it is code to do big number integer arithmetic that is 
optimised for sun4u to explicitly (ab)using the FPU.  This isn''t 
guessing it is was a deliberate design choice.

Specifically this code here:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/common/bignum/sun4u/

Note that there are separate kernel and user land variants of that.

-- 
Darren J Moffat

On 23/12/2010 15:18, Garrett D''Amore wrote:
> Thanks for the clarification. I guess I need to go back and figure out
> how ZFS crypto keying is performed. I guess most likely the key is
> generated from some sort of one-way hash from a passphrase?
See http://blogs.sun.com/darren/entry/zfs_encryption_what_is_on where I 
explain all the type of keys used and how they are generated as well as 
how passphrases are turned into AES wrapping keys (using PKCS#5 PBE).

-- 
Darren J Moffat