llvm dev - May 2016 - problems with objects larger than PTRDIFF_MAX

On 2016-05-20 19:58, David Majnemer via llvm-dev wrote:
> I've come across this issue before and came to the following
conclusion:
> - We are not obligated to support objects that large, C11 5.2.4.1/1 only
> requires that we support objects of size 65535!
Right, the standard doesn't require it. But I guess you don't imply that
it's fine for clang to silently miscompile any program that works with 
objects larger than 65535 bytes.
> Their guidance for maximum
> object size is stated to be half of SIZE_MAX in C11 K.3.4/4 which is
> typically equivalent to PTRDIFF_MAX.
Whose guidance? Annex K is kinda alien to the rest of the standard and 
its future is not clear. See, e.g., 
http://open-std.org/jtc1/sc22/wg14/www/docs/n1969.htm .
> - The expectation that PTRDIFF_MAX is more or less a proxy for the largest
> object size is not uncommon.   For example, C++'s std::count
doesn't return
> a size_t but a iterator_traits<>::difference_type which is going to
be a
> ptrdiff_t for things like std::vector.
Bug in C++? Because max_size() of std::vector<char> has the type 
`size_type` and returns SIZE_MAX in practice (even on x86-64).

Let's see other examples (on x86-32):

- glibc's malloc is happy to allocate more than PTRDIFF_MAX bytes (and 
this is important in practice so (at least some) glibc devs are 
reluctant to "fix" it);

- clang has a compile-time limit for sizes of objects and types which is 
SIZE_MAX. For example, it compiles "char a[-1ul]; int main() {}" but 
complains about "char a[-1ul + 1ull]; int main() {}";

- `new` for more than PTRDIFF_MAX works fine when compiled by clang++ 
(but throws std::bad_array_new_length when compiled by g++).
> On Fri, May 20, 2016 at 1:31 AM, John Regehr via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> It could be that 32-bit systems are disappearing so rapidly that nobody
>> cares too much about this issue,
Please note that there is a whole new 32-bit architecture -- x32.
>> but this blog post is still worth reading:
>>
>>    http://trust-in-soft.com/objects-larger-than-ptrdiff_max-bytes/
-- 
Alexander Cherepanov

On Sun, May 29, 2016 at 6:18 AM, Alexander Cherepanov <ch3root at
openwall.com>
wrote:
> On 2016-05-20 19:58, David Majnemer via llvm-dev wrote:
>
>> I've come across this issue before and came to the following
conclusion:
>> - We are not obligated to support objects that large, C11 5.2.4.1/1
only
>> requires that we support objects of size 65535!
>>
>
> Right, the standard doesn't require it. But I guess you don't imply
that
> it's fine for clang to silently miscompile any program that works with
> objects larger than 65535 bytes.

I think that we should error when we can statically determine that an
object is larger than PTRDIFF_MAX.

>
>
> Their guidance for maximum
>> object size is stated to be half of SIZE_MAX in C11 K.3.4/4 which is
>> typically equivalent to PTRDIFF_MAX.
>>
>
> Whose guidance? Annex K is kinda alien to the rest of the standard and its
> future is not clear. See, e.g.,
> http://open-std.org/jtc1/sc22/wg14/www/docs/n1969.htm .

That document objects to much of Annex K but doesn't say anything about
K.3.4/4.

>
>
> - The expectation that PTRDIFF_MAX is more or less a proxy for the largest
>> object size is not uncommon.   For example, C++'s std::count
doesn't
>> return
>> a size_t but a iterator_traits<>::difference_type which is going
to be a
>> ptrdiff_t for things like std::vector.
>>
>
> Bug in C++? Because max_size() of std::vector<char> has the type
> `size_type` and returns SIZE_MAX in practice (even on x86-64).
>
Some folks in the C++ community believe that using unsigned types in those
places was the wrong decision.

>
> Let's see other examples (on x86-32):
>
> - glibc's malloc is happy to allocate more than PTRDIFF_MAX bytes (and
> this is important in practice so (at least some) glibc devs are reluctant
> to "fix" it);
>
glibc is not the only game in town.
Android prevents it: https://android-review.googlesource.com/#/c/170800/
jemalloc prevents it:
https://github.com/jemalloc/jemalloc/commit/0c516a00c4cb28cff55ce0995f756b5aae074c9e

There are other implementations of malloc which also act similarly.

> - clang has a compile-time limit for sizes of objects and types which is
> SIZE_MAX. For example, it compiles "char a[-1ul]; int main() {}"
but
> complains about "char a[-1ul + 1ull]; int main() {}";
>
> - `new` for more than PTRDIFF_MAX works fine when compiled by clang++ (but
> throws std::bad_array_new_length when compiled by g++).
>
> On Fri, May 20, 2016 at 1:31 AM, John Regehr via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> It could be that 32-bit systems are disappearing so rapidly that nobody
>>> cares too much about this issue,
>>>
>>
> Please note that there is a whole new 32-bit architecture -- x32.
>
>
> but this blog post is still worth reading:
>>>
>>>    http://trust-in-soft.com/objects-larger-than-ptrdiff_max-bytes/
>>>
>>
> --
> Alexander Cherepanov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160529/5659d353/attachment.html>

On 2016-05-29 21:16, David Majnemer wrote:
> On Sun, May 29, 2016 at 6:18 AM, Alexander Cherepanov <ch3root at
openwall.com>
> wrote:
>
>> On 2016-05-20 19:58, David Majnemer via llvm-dev wrote:
>>
>>> I've come across this issue before and came to the following
conclusion:
>>> - We are not obligated to support objects that large, C11 5.2.4.1/1
only
>>> requires that we support objects of size 65535!
>>>
>>
>> Right, the standard doesn't require it. But I guess you don't
imply that
>> it's fine for clang to silently miscompile any program that works
with
>> objects larger than 65535 bytes.
>
> I think that we should error when we can statically determine that an
> object is larger than PTRDIFF_MAX.
It seems we agree that the current situation is not good and something 
has to be fixed in clang/llvm. Ok.

Why don't you think that clang/llvm should just be fixed to support 
objects larger than PTRDIFF_MAX bytes?
>> - The expectation that PTRDIFF_MAX is more or less a proxy for the
largest
>>> object size is not uncommon.   For example, C++'s std::count
doesn't
>>> return
>>> a size_t but a iterator_traits<>::difference_type which is
going to be a
>>> ptrdiff_t for things like std::vector.
>>
>> Bug in C++? Because max_size() of std::vector<char> has the type
>> `size_type` and returns SIZE_MAX in practice (even on x86-64).
>
> Some folks in the C++ community believe that using unsigned types in those
> places was the wrong decision.
Result of max_size() cannot be changed without changing its type?
>> Let's see other examples (on x86-32):
>>
>> - glibc's malloc is happy to allocate more than PTRDIFF_MAX bytes
(and
>> this is important in practice so (at least some) glibc devs are
reluctant
>> to "fix" it);
>>
>
> glibc is not the only game in town.
Right, for example musl doesn't create objects larger than PTRDIFF_MAX 
bytes. I think this is good that different libc implementations are free 
to choose their own policies.

The problem is that this is only possible when compilers support large 
objects. When compilers don't support large objects (like now), support 
in libc is not enough -- you get your large objects but your accesses to 
it will be miscompiled. Sometimes.
> Android prevents it: https://android-review.googlesource.com/#/c/170800/
> jemalloc prevents it:
>
https://github.com/jemalloc/jemalloc/commit/0c516a00c4cb28cff55ce0995f756b5aae074c9e
A good share of the reasons for this is exactly the fact that clang/llvm 
don't support large objects. So this reasoning is circular:-)
> There are other implementations of malloc which also act similarly.
-- 
Alexander Cherepanov