llvm dev - Jun 2014 - [LLVMdev] -fvisibility=hidden, and typeinfo, and type-erasure

[Was initially posted on cfe-users, sorry.]

Hi,

I'm sorry my message is quite long, the TL;DR version is "g++ and
clang++ seem to have different opinions on how RTTI, templates, and ELF
visibility should interact".



I can't tell whether this is a bug or not: I have found no relevant
documentation that could help me decide whether this behavior is meant, or not. 
All I can say is that the current behavior is not the one I would expect, but
maybe you guys have a different opinion, which I'd be happy to hear about. 
To my eyes it looks like a violation of the One Definition Rule, but since ELF
visibility issues are not covered by the standard, this is wishful thinking :)

I'm reproducing here basically what I had written there:

    http://stackoverflow.com/questions/19496643/

-------------------------------------------------

This is a scaled down version of a problem I am facing with clang++ on Mac OS X.

The failure
==========
I have this big piece of software in C++ with a large set of symbols in the
object files, so I'm using `-fvisibility=hidden` to keep my symbol tables
small.  It is well known that in such a case one must pay extra attention to the
vtables, and I suppose I face this problem.  I don't know however, how to
address it elegantly in a way that pleases both gcc and clang.

Consider a `base` class which features a down-casting operator, `as`, and a
`derived` class template, that contains some payload.  The pair
`base`/`derived<T>` is used to implement type-erasure:

   // foo.hh

   #define API __attribute__((visibility("default")))

   struct API base
   {
     virtual ~base() {}

     template <typename T>
     const T& as() const
     {
       return dynamic_cast<const T&>(*this);
     }
   };

   template <typename T>
   struct API derived: base
   {};

   struct payload {}; // *not* flagged as "default visibility".

   API void bar(const base& b);
   API void baz(const base& b);


Then I have two different compilation units that provide a similar service,
which I can approximate as twice the same feature: down-casting from `base` to
`derive<payload>`:

   // bar.cc
   #include "foo.hh"
   void bar(const base& b)
   {
     b.as<derived<payload>>();
   }

and

   // baz.cc
   #include "foo.hh"
   void baz(const base& b)
   {
     b.as<derived<payload>>();
   }
>From these two files, I build a dylib.  Here is the `main` function, calling
these functions from the dylib:
   // main.cc
   #include <stdexcept>
   #include <iostream>
   #include "foo.hh"

   int main()
   try
     {
       derived<payload> d;
       bar(d);
       baz(d);
     }
   catch (std::exception& e)
     {
       std::cerr << e.what() << std::endl;
     }

Finally, a Makefile to compile and link everybody.  Nothing special here,
except, of course, `-fvisibility=hidden`.

   CXX = clang++
   CXXFLAGS = -std=c++11 -fvisibility=hidden

   all: main

   main: main.o bar.dylib baz.dylib
   	$(CXX) -o $@ $^

   %.dylib: %.cc foo.hh
   	$(CXX) $(CXXFLAGS) -shared -o $@ $<

   %.o: %.cc foo.hh
   	$(CXX) $(CXXFLAGS) -c -o $@ $<

   clean:
   	rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib

The run succeeds with gcc (4.8 and 4.9) on OS X:

   $ make clean && make CXX=g++-mp-4.8 && ./main 
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -c main.cc -o main.o
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   g++-mp-4.8 -o main main.o bar.dylib baz.dylib

However with clang (3.4 and 3.5), this fails (the typeids have different
addresses):

   $ make clean && make CXX=clang++-mp-3.4 && ./main
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c main.cc -o main.o
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
   std::bad_cast

However it works if I tag my payload with a public visibility:

   struct API payload {};

but I do not want to expose the payload type.  So my questions are:

1. why are GCC and Clang different here?
2. is it _really_ working with GCC, or I was just "lucky" in my use of
undefined behavior?
3. do I have a means to avoid making `payload` go public with Clang++?

Thanks in advance.

Type equality of visible class templates with invisible type parameters (EDIT)
=============================================================================
I have now a better understanding of what is happening.  It is appears that both
GCC _and_ clang require both the class template and its parameter to be visible
(in the ELF sense) to build a unique type.  If you change the `bar.cc` and
`baz.cc` functions as follows:

   // bar.cc
   #include "foo.hh"
   void bar(const base& b)
   {
     std::cerr
       << "bar value: " << &typeid(b) <<
std::endl
       << "bar type:  " <<
&typeid(derived<payload>) << std::endl
       << "bar equal: " << (typeid(b) ==
typeid(derived<payload>)) << std::endl;
     b.as<derived<payload>>();
   }

and *if* you make `payload` visible too:

   struct API payload {};

then both GCC and Clang succeed (same typeid address):

   $ make clean && make CXX=g++-mp-4.8
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   ./g++-mp-4.8 -o main main.o bar.dylib baz.dylib
   $ ./main
   bar value: 0x106785140
   bar type:  0x106785140
   bar equal: 1
   baz value: 0x106785140
   baz type:  0x106785140
   baz equal: 1

   $ make clean && make CXX=clang++-mp-3.4
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
   $ ./main
   bar value: 0x10a6d5110
   bar type:  0x10a6d5110
   bar equal: 1
   baz value: 0x10a6d5110
   baz type:  0x10a6d5110
   baz equal: 1

Type equality is easy to check, there is actually a single instantiation of the
type, as witnessed by its unique address.

However, if you remove the visible attribute from `payload`:

   struct payload {};

then you get with GCC:

   $ make clean && make CXX=g++-mp-4.8
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   g++-mp-4.8 -o main main.o bar.dylib baz.dylib
   $ ./main
   bar value: 0x10faea120
   bar type:  0x10faf1090
   bar equal: 1
   baz value: 0x10faea120
   baz type:  0x10fafb090
   baz equal: 1

Now there are several instantiation of the type `derived<payload>` (as
witnessed by the three different addresses), but GCC sees these types are equal,
and (of course) the two `dynamic_cast` pass.

In the case of clang, it's different:

   $ make clean && make CXX=clang++-mp-3.4
   rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
   clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
   .clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
   $ ./main
   bar value: 0x1012ae0f0
   bar type:  0x1012b3090
   bar equal: 0
   std::bad_cast

There are also three instantiations of the type (removing the failing
`dynamic_cast` does show that there are three), but this time, they are not
equal, and the `dynamic_cast` (of course) fails.

Now the question turns into:
1. is this difference between both compilers wanted by their authors
2. if not, what is "expected" behavior between both

I prefer GCC's semantics, as it allows to really implement type-erasure
without any need to expose publicly the wrapped types.

I think the difference is actually in the c++ library. It looks like
libstdc++ changed to always use strcmp of the typeinfo names:

https://gcc.gnu.org/viewcvs/gcc?view=revision&revision=149964

Should we do the same with libc++?


On 2 June 2014 03:07, Akim Demaille <akim.demaille at gmail.com>
wrote:
> [Was initially posted on cfe-users, sorry.]
>
> Hi,
>
> I'm sorry my message is quite long, the TL;DR version is "g++ and
clang++ seem to have different opinions on how RTTI, templates, and ELF
visibility should interact".
>
>
>
> I can't tell whether this is a bug or not: I have found no relevant
documentation that could help me decide whether this behavior is meant, or not. 
All I can say is that the current behavior is not the one I would expect, but
maybe you guys have a different opinion, which I'd be happy to hear about. 
To my eyes it looks like a violation of the One Definition Rule, but since ELF
visibility issues are not covered by the standard, this is wishful thinking :)
>
> I'm reproducing here basically what I had written there:
>
>     http://stackoverflow.com/questions/19496643/
>
> -------------------------------------------------
>
> This is a scaled down version of a problem I am facing with clang++ on Mac
OS X.
>
> The failure
> ==========>
> I have this big piece of software in C++ with a large set of symbols in the
object files, so I'm using `-fvisibility=hidden` to keep my symbol tables
small.  It is well known that in such a case one must pay extra attention to the
vtables, and I suppose I face this problem.  I don't know however, how to
address it elegantly in a way that pleases both gcc and clang.
>
> Consider a `base` class which features a down-casting operator, `as`, and a
`derived` class template, that contains some payload.  The pair
`base`/`derived<T>` is used to implement type-erasure:
>
>    // foo.hh
>
>    #define API __attribute__((visibility("default")))
>
>    struct API base
>    {
>      virtual ~base() {}
>
>      template <typename T>
>      const T& as() const
>      {
>        return dynamic_cast<const T&>(*this);
>      }
>    };
>
>    template <typename T>
>    struct API derived: base
>    {};
>
>    struct payload {}; // *not* flagged as "default visibility".
>
>    API void bar(const base& b);
>    API void baz(const base& b);
>
>
> Then I have two different compilation units that provide a similar service,
which I can approximate as twice the same feature: down-casting from `base` to
`derive<payload>`:
>
>    // bar.cc
>    #include "foo.hh"
>    void bar(const base& b)
>    {
>      b.as<derived<payload>>();
>    }
>
> and
>
>    // baz.cc
>    #include "foo.hh"
>    void baz(const base& b)
>    {
>      b.as<derived<payload>>();
>    }
>
> From these two files, I build a dylib.  Here is the `main` function,
calling these functions from the dylib:
>
>    // main.cc
>    #include <stdexcept>
>    #include <iostream>
>    #include "foo.hh"
>
>    int main()
>    try
>      {
>        derived<payload> d;
>        bar(d);
>        baz(d);
>      }
>    catch (std::exception& e)
>      {
>        std::cerr << e.what() << std::endl;
>      }
>
> Finally, a Makefile to compile and link everybody.  Nothing special here,
except, of course, `-fvisibility=hidden`.
>
>    CXX = clang++
>    CXXFLAGS = -std=c++11 -fvisibility=hidden
>
>    all: main
>
>    main: main.o bar.dylib baz.dylib
>         $(CXX) -o $@ $^
>
>    %.dylib: %.cc foo.hh
>         $(CXX) $(CXXFLAGS) -shared -o $@ $<
>
>    %.o: %.cc foo.hh
>         $(CXX) $(CXXFLAGS) -c -o $@ $<
>
>    clean:
>         rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>
> The run succeeds with gcc (4.8 and 4.9) on OS X:
>
>    $ make clean && make CXX=g++-mp-4.8 && ./main
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -c main.cc -o main.o
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
>    g++-mp-4.8 -o main main.o bar.dylib baz.dylib
>
> However with clang (3.4 and 3.5), this fails (the typeids have different
addresses):
>
>    $ make clean && make CXX=clang++-mp-3.4 && ./main
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c main.cc -o main.o
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib
bar.cc
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib
baz.cc
>    clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
>    std::bad_cast
>
> However it works if I tag my payload with a public visibility:
>
>    struct API payload {};
>
> but I do not want to expose the payload type.  So my questions are:
>
> 1. why are GCC and Clang different here?
> 2. is it _really_ working with GCC, or I was just "lucky" in my
use of undefined behavior?
> 3. do I have a means to avoid making `payload` go public with Clang++?
>
> Thanks in advance.
>
> Type equality of visible class templates with invisible type parameters
(EDIT)
>
=============================================================================>
> I have now a better understanding of what is happening.  It is appears that
both GCC _and_ clang require both the class template and its parameter to be
visible (in the ELF sense) to build a unique type.  If you change the `bar.cc`
and `baz.cc` functions as follows:
>
>    // bar.cc
>    #include "foo.hh"
>    void bar(const base& b)
>    {
>      std::cerr
>        << "bar value: " << &typeid(b) <<
std::endl
>        << "bar type:  " <<
&typeid(derived<payload>) << std::endl
>        << "bar equal: " << (typeid(b) ==
typeid(derived<payload>)) << std::endl;
>      b.as<derived<payload>>();
>    }
>
> and *if* you make `payload` visible too:
>
>    struct API payload {};
>
> then both GCC and Clang succeed (same typeid address):
>
>    $ make clean && make CXX=g++-mp-4.8
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
>    ./g++-mp-4.8 -o main main.o bar.dylib baz.dylib
>    $ ./main
>    bar value: 0x106785140
>    bar type:  0x106785140
>    bar equal: 1
>    baz value: 0x106785140
>    baz type:  0x106785140
>    baz equal: 1
>
>    $ make clean && make CXX=clang++-mp-3.4
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib
bar.cc
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib
baz.cc
>    clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
>    $ ./main
>    bar value: 0x10a6d5110
>    bar type:  0x10a6d5110
>    bar equal: 1
>    baz value: 0x10a6d5110
>    baz type:  0x10a6d5110
>    baz equal: 1
>
> Type equality is easy to check, there is actually a single instantiation of
the type, as witnessed by its unique address.
>
> However, if you remove the visible attribute from `payload`:
>
>    struct payload {};
>
> then you get with GCC:
>
>    $ make clean && make CXX=g++-mp-4.8
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o bar.dylib bar.cc
>    g++-mp-4.8 -std=c++11 -fvisibility=hidden -shared -o baz.dylib baz.cc
>    g++-mp-4.8 -o main main.o bar.dylib baz.dylib
>    $ ./main
>    bar value: 0x10faea120
>    bar type:  0x10faf1090
>    bar equal: 1
>    baz value: 0x10faea120
>    baz type:  0x10fafb090
>    baz equal: 1
>
> Now there are several instantiation of the type `derived<payload>`
(as witnessed by the three different addresses), but GCC sees these types are
equal, and (of course) the two `dynamic_cast` pass.
>
> In the case of clang, it's different:
>
>    $ make clean && make CXX=clang++-mp-3.4
>    rm -f main main.o bar.o baz.o bar.dylib baz.dylib libba.dylib
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -c -o main.o main.cc
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o bar.dylib
bar.cc
>    clang++-mp-3.4 -std=c++11 -fvisibility=hidden -shared -o baz.dylib
baz.cc
>    .clang++-mp-3.4 -o main main.o bar.dylib baz.dylib
>    $ ./main
>    bar value: 0x1012ae0f0
>    bar type:  0x1012b3090
>    bar equal: 0
>    std::bad_cast
>
> There are also three instantiations of the type (removing the failing
`dynamic_cast` does show that there are three), but this time, they are not
equal, and the `dynamic_cast` (of course) fails.
>
> Now the question turns into:
> 1. is this difference between both compilers wanted by their authors
> 2. if not, what is "expected" behavior between both
>
> I prefer GCC's semantics, as it allows to really implement type-erasure
without any need to expose publicly the wrapped types.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Wed, Jun 04, 2014 at 06:32:35PM -0400, Rafael Espíndola
wrote:
> I think the difference is actually in the c++ library. It looks like
> libstdc++ changed to always use strcmp of the typeinfo names:
> 
> https://gcc.gnu.org/viewcvs/gcc?view=revision&revision=149964
> 
> Should we do the same with libc++?
No. It pessimizes correctly behaving code to work around code that
violates the ELF ABI. Consider an application that loads plugins with
RTLD_LOCAL. Two plugins implement a (hidden) type called "Plugin". The
GCC change now makes this completely separate types identical for all
RTTI purposes. That makes no sense at all.

Joerg

Le 5 juin 2014 à 00:32, Rafael Espíndola <rafael.espindola at gmail.com> a
écrit :
> I think the difference is actually in the c++ library. It looks like
> libstdc++ changed to always use strcmp of the typeinfo names:
> 
> https://gcc.gnu.org/viewcvs/gcc?view=revision&revision=149964
> 
> Should we do the same with libc++?
What do people think about this issue?