llvm dev - Mar 2016 - Incorrect return values for APFloat::convertFromString?

I noticed some odd behavior with APFloat's convertFromString method.


1.       If I pass the hex representation of the closest value to 0.1
(0x19999Ap-24), everything is fine and opOk is returned. However, if I pass the
same value as a decimal string (0.10000002384185791015625), opInexact is set.

2.       On the lower end of the scale, the smallest denormal 0x1p-149 returns
opOk, but the slightly larger value 0x1.8p-149 has opUnderflow set and (as
expected) opInexact.

Is this behavior correct?

--
Johannes S. Mueller-Roemer, MSc
Wiss. Mitarbeiter - Interactive Engineering Technologies (IET)

Fraunhofer-Institut für Graphische Datenverarbeitung IGD
Fraunhoferstr. 5  |  64283 Darmstadt  |  Germany
Tel +49 6151 155-606  |  Fax +49 6151 155-139
johannes.mueller-roemer at igd.fraunhofer.de  |  www.igd.fraunhofer.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160302/31a6c624/attachment.html>

On Wed, Mar 02, 2016 at 03:47:43PM +0000, Mueller-Roemer, Johannes Sebastian via
llvm-dev wrote:
> 1.       If I pass the hex representation of the closest value to 0.1
> (0x19999Ap-24), everything is fine and opOk is returned. However, if I
> pass the same value as a decimal string (0.10000002384185791015625),
> opInexact is set.
Well, they are not the same value. The decimal string depends on the
mantissa size and the rounding mode. For the hex version, only the
mantissa size has to be large enough to not require truncation/rounding.

Joerg

The mantissa size is specified at initialization of the APFloat and the rounding
mode shouldn't matter in determining *if* a value was rounded. The strings
themselves are exact representations of the same value.

--
Johannes S. Mueller-Roemer, MSc
Wiss. Mitarbeiter - Interactive Engineering Technologies (IET)

Fraunhofer-Institut für Graphische Datenverarbeitung IGD
Fraunhoferstr. 5  |  64283 Darmstadt  |  Germany
Tel +49 6151 155-606  |  Fax +49 6151 155-139
johannes.mueller-roemer at igd.fraunhofer.de  |  www.igd.fraunhofer.de


-----Original Message-----
From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of Joerg
Sonnenberger via llvm-dev
Sent: Wednesday, March 02, 2016 17:03
To: llvm-dev at lists.llvm.org
Subject: Re: [llvm-dev] Incorrect return values for APFloat::convertFromString?

On Wed, Mar 02, 2016 at 03:47:43PM +0000, Mueller-Roemer, Johannes Sebastian via
llvm-dev wrote:
> 1.       If I pass the hex representation of the closest value to 0.1
> (0x19999Ap-24), everything is fine and opOk is returned. However, if I 
> pass the same value as a decimal string (0.10000002384185791015625), 
> opInexact is set.
Well, they are not the same value. The decimal string depends on the mantissa
size and the rounding mode. For the hex version, only the mantissa size has to
be large enough to not require truncation/rounding.

Joerg
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev