llvm dev - Feb 2021 - [libcxx-dev] Shipping custom libc++ on MacOS

> On Feb 1, 2021, at 15:16, Louis Dionne via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> 
> 
>> On Feb 1, 2021, at 02:37, Tobias Hieta via libcxx-dev <libcxx-dev at
lists.llvm.org> wrote:
>> 
>> Hello,
>> 
>> We are working on an application where we want to have full support
>> for C++17 but still ship on older versions of macOS (10.9/10.10 in
>> this case). Our plan was to build our own libc++ from llvm and ship it
>> with the application and then pass
"-D_LIBCPP_DISABLE_AVAILABILITY"
>> 
>> This seemed to work fine in internal testing - but then we ran into a
>> similar issue as the one described here:
>> https://reviews.llvm.org/D74489#2497044
>> 
>> Now I can patch my copy of libc++ with the upstream fix for this - but
>> the comment below made me think otherwise:
>> https://reviews.llvm.org/D74489#2505156
>> 
>> We have run into problem where if you ship your own libc++ the system
>> frameworks loads the system libc++ and some strange errors happen - we
>> thought we could work around these issues - but maybe this is just us
>> being naive.
>> 
>> So I guess my question is - is it possible to ship a custom libc++ in
>> a .app archive for older versions of macOS in order to use C++17? Or
>> is my only hope to raise the minimum version of our app and always
>> link to the system libc++?
> 
> Generally, I would say this is a tricky thing to do. As you mention, the
problem is that if you link against any other library that does link against the
system-provided libc++.dylib, you'll have ODR violations because some
symbols will be found in both libraries. As a result, I believe you could also
end up in a situation where you end up using globals from one library when the
globals from the other library has been initialized. So basically, nothing good
happens.
> 
> If you can ensure that your application doesn't pull in the system
libc++.dylib (even transitively), then this would in theory be safe to do. Due
to the brittleness of the solution, my immediate reaction would be to not
recommend that. The reason why some parts of C++17 aren't available on older
platforms is that we put the vtables required for exception classes in the
dylib. That is a code size optimization that we think is worth the trouble. You
can generally use the parts of C++17 APIs that can't throw exceptions even
on older platforms because they don't require those vtable symbols - perhaps
that is something you can consider?
Also, Chrome is known to do something similar to what you're inquiring
about. They build libc++ statically with hidden visibility, and then they link
it into their application. I assume they also make sure to not export any symbol
from libc++ that would be created by using the libc++ headers (e.g. by
instantiating some function template). I don't know their setup very well,
but I assume they make sure to not link against any other system library that
does require libc++.dylib from the system, or else things get trickier as I
explained above. But as long as you satisfy that requirements, I believe things
have worked fairly well for them so far. You just have to be able and willing to
walk that fine line.

Louis

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210201/62cdc81e/attachment.html>

On MacPorts we are just getting into this issue, in our case trying to build a
new libc++.dylib that will be accepted by the existing system frameworks on 10.7
to 10.13 to support c++17 software on older systems without resorting to
libstdc++ from gcc.

Is this macro of value in helping avoid ODR violations?  I’m just investigating
that now:

 _LIBCPP_HIDE_FROM_ABI_PER_TU

Ken


> On Feb 1, 2021, at 12:19 PM, Louis Dionne via libcxx-dev <libcxx-dev at
lists.llvm.org> wrote:
> 
> 
> 
>> On Feb 1, 2021, at 15:16, Louis Dionne via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> 
>> 
>>> On Feb 1, 2021, at 02:37, Tobias Hieta via libcxx-dev
<libcxx-dev at lists.llvm.org <mailto:libcxx-dev at lists.llvm.org>>
wrote:
>>> 
>>> Hello,
>>> 
>>> We are working on an application where we want to have full support
>>> for C++17 but still ship on older versions of macOS (10.9/10.10 in
>>> this case). Our plan was to build our own libc++ from llvm and ship
it
>>> with the application and then pass
"-D_LIBCPP_DISABLE_AVAILABILITY"
>>> 
>>> This seemed to work fine in internal testing - but then we ran into
a
>>> similar issue as the one described here:
>>> https://reviews.llvm.org/D74489#2497044
<https://reviews.llvm.org/D74489#2497044>
>>> 
>>> Now I can patch my copy of libc++ with the upstream fix for this -
but
>>> the comment below made me think otherwise:
>>> https://reviews.llvm.org/D74489#2505156
>>> 
>>> We have run into problem where if you ship your own libc++ the
system
>>> frameworks loads the system libc++ and some strange errors happen -
we
>>> thought we could work around these issues - but maybe this is just
us
>>> being naive.
>>> 
>>> So I guess my question is - is it possible to ship a custom libc++
in
>>> a .app archive for older versions of macOS in order to use C++17?
Or
>>> is my only hope to raise the minimum version of our app and always
>>> link to the system libc++?
>> 
>> Generally, I would say this is a tricky thing to do. As you mention,
the problem is that if you link against any other library that does link against
the system-provided libc++.dylib, you'll have ODR violations because some
symbols will be found in both libraries. As a result, I believe you could also
end up in a situation where you end up using globals from one library when the
globals from the other library has been initialized. So basically, nothing good
happens.
>> 
>> If you can ensure that your application doesn't pull in the system
libc++.dylib (even transitively), then this would in theory be safe to do. Due
to the brittleness of the solution, my immediate reaction would be to not
recommend that. The reason why some parts of C++17 aren't available on older
platforms is that we put the vtables required for exception classes in the
dylib. That is a code size optimization that we think is worth the trouble. You
can generally use the parts of C++17 APIs that can't throw exceptions even
on older platforms because they don't require those vtable symbols - perhaps
that is something you can consider?
> 
> Also, Chrome is known to do something similar to what you're inquiring
about. They build libc++ statically with hidden visibility, and then they link
it into their application. I assume they also make sure to not export any symbol
from libc++ that would be created by using the libc++ headers (e.g. by
instantiating some function template). I don't know their setup very well,
but I assume they make sure to not link against any other system library that
does require libc++.dylib from the system, or else things get trickier as I
explained above. But as long as you satisfy that requirements, I believe things
have worked fairly well for them so far. You just have to be able and willing to
walk that fine line.
> 
> Louis
> 
> _______________________________________________
> libcxx-dev mailing list
> libcxx-dev at lists.llvm.org <mailto:libcxx-dev at lists.llvm.org>
> https://lists.llvm.org/cgi-bin/mailman/listinfo/libcxx-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/libcxx-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210201/5b8f774b/attachment.html>

We use this approach in Fuchsia as well where we always statically link
libc++ into our host tools and we haven't encountered any issues, but these
are command line development tools which generally don't use any frameworks
or host libraries. The only minor problem we ran into is that Clang's
Darwin driver doesn't support `-static-libstdc++` flag, so we instead have
to pass `-nostdlib++ ${clang_prefix}/../lib/libc++.a` through linker flags
which is a bit unfortunate but not a big deal.

On Mon, Feb 1, 2021 at 12:19 PM Louis Dionne via libcxx-dev <
libcxx-dev at lists.llvm.org> wrote:
>
>
> On Feb 1, 2021, at 15:16, Louis Dionne via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
>
> On Feb 1, 2021, at 02:37, Tobias Hieta via libcxx-dev <
> libcxx-dev at lists.llvm.org> wrote:
>
> Hello,
>
> We are working on an application where we want to have full support
> for C++17 but still ship on older versions of macOS (10.9/10.10 in
> this case). Our plan was to build our own libc++ from llvm and ship it
> with the application and then pass
"-D_LIBCPP_DISABLE_AVAILABILITY"
>
> This seemed to work fine in internal testing - but then we ran into a
> similar issue as the one described here:
> https://reviews.llvm.org/D74489#2497044
>
> Now I can patch my copy of libc++ with the upstream fix for this - but
> the comment below made me think otherwise:
> https://reviews.llvm.org/D74489#2505156
>
> We have run into problem where if you ship your own libc++ the system
> frameworks loads the system libc++ and some strange errors happen - we
> thought we could work around these issues - but maybe this is just us
> being naive.
>
> So I guess my question is - is it possible to ship a custom libc++ in
> a .app archive for older versions of macOS in order to use C++17? Or
> is my only hope to raise the minimum version of our app and always
> link to the system libc++?
>
>
> Generally, I would say this is a tricky thing to do. As you mention, the
> problem is that if you link against any other library that does link
> against the system-provided libc++.dylib, you'll have ODR violations
> because some symbols will be found in both libraries. As a result, I
> believe you could also end up in a situation where you end up using globals
> from one library when the globals from the other library has been
> initialized. So basically, nothing good happens.
>
> If you can ensure that your application doesn't pull in the system
> libc++.dylib (even transitively), then this would in theory be safe to do.
> Due to the brittleness of the solution, my immediate reaction would be to
> not recommend that. The reason why some parts of C++17 aren't available
on
> older platforms is that we put the vtables required for exception classes
> in the dylib. That is a code size optimization that we think is worth the
> trouble. You can generally use the parts of C++17 APIs that can't throw
> exceptions even on older platforms because they don't require those
vtable
> symbols - perhaps that is something you can consider?
>
>
> Also, Chrome is known to do something similar to what you're inquiring
> about. They build libc++ statically with hidden visibility, and then they
> link it into their application. I assume they also make sure to not export
> any symbol from libc++ that would be created by using the libc++ headers
> (e.g. by instantiating some function template). I don't know their
setup
> very well, but I assume they make sure to not link against any other system
> library that does require libc++.dylib from the system, or else things get
> trickier as I explained above. But as long as you satisfy that
> requirements, I believe things have worked fairly well for them so far. You
> just have to be able and willing to walk that fine line.
>
> Louis
>
> _______________________________________________
> libcxx-dev mailing list
> libcxx-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/libcxx-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210202/d5652dae/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210202/d5652dae/attachment.bin>