llvm dev - Mar 2013 - [LLVMdev] Feedback required on proper dllexport/import implementation

Hello,

while improving and extending support for dllexport/import I have
noticed that the current way these are implemented is problematic and I
would like some input on how to proceed.

Currently dllexport/dllimport is treated as linkage type. This conflicts
with inlined functions because there is no linkage for the combination
of both. On first though, combining both doesn't make sense, but take
this class as an example:

  struct __declspec(dllexport/dllimport) X {
    X() {}
  };

Such constructs with empty or simple inline functions can be found for
example in Microsoft headers or even libc++ headers.

Ignoring the dllexport/import attribute for such functions would seem
most sensible (and can be implemented easily), but MSVC does the
opposite: For imported inline functions the function body is dropped and
__imp_ calls are emitted, and exported inline functions are placed into
COMDAT sections. The latter cannot be expressed because it requires
linkonce_odr and dllexport linkage.

The question now is how to implement this. After a brief discussion, I
can think of four ways:

1. Add additional linkage type(s) for the combinations to
   GlobalValue::LinkageTypes.

   This appears to be the least invasive way, but adds new linkage types
   to an already large list.

2. Handle dllexport/import similar to ELF visibility by adding new
   "visibility" types to GlobalValue::VisibilityTypes and IR
visibility
   styles.

   This feels like kind of a band-aid. While dllexport could be
   construed as similar to default visibility (some code uses both in
   the same place depending on platform), dllimport feels wrong here.
   This would also prevent mixing ELF visibility with dllexport/import.

   The size of GlobalValue can be kept the same by simply adjusting the
   bit-fields for linkage and visibility.

3. Add a new enum for dllimport/export to GlobalValue and IR global
   variables and functions, similar to ELF visibility.

   This is similar to (2.), without the awkward piggybacking on
   visibility. But it requires extensions to IR just for a single
   platform.

   The size of GlobalValue can be kept the same by simply adjusting the
   bit-fields.

4. Handle dllexport/import as platform-specific IR function attributes.

   Because dllexport/import can also apply to globals which have no
   attributes, this requires keeping the dllexport/import linkage types
   just for them.

   Inline does not apply to globals, but MSVC can actually produce
   initialized dllexported globals, placed in COMDAT sections, with
   __declspec(selectany). I have no idea if anyone actually does this.
   LLVM also does not support __declspec(selectany) yet.

There may be even more or better ways to implement this. It may be good
to keep a future implementation of __declspec(selectany) in mind when
thinking about this issue.


-Nico

On Mon, Mar 25, 2013 at 9:43 PM, Nico Rieck <nico.rieck at gmail.com>
wrote:
> Hello,
>
> while improving and extending support for dllexport/import I have
> noticed that the current way these are implemented is problematic and I
> would like some input on how to proceed.
>
> Currently dllexport/dllimport is treated as linkage type. This conflicts
> with inlined functions because there is no linkage for the combination
> of both. On first though, combining both doesn't make sense, but take
> this class as an example:
>
>   struct __declspec(dllexport/dllimport) X {
>     X() {}
>   };
>
> Such constructs with empty or simple inline functions can be found for
> example in Microsoft headers or even libc++ headers.
>
> Ignoring the dllexport/import attribute for such functions would seem
> most sensible (and can be implemented easily), but MSVC does the
> opposite: For imported inline functions the function body is dropped and
> __imp_ calls are emitted, and exported inline functions are placed into
> COMDAT sections. The latter cannot be expressed because it requires
> linkonce_odr and dllexport linkage.
>
Hang on, to me the docs seem to say the dllimport case is slightly
different:
http://msdn.microsoft.com/en-us/library/xa0d9ste.aspx

dllexport: May be inlined, but always gets instantiated as COMDAT and
ultimately gets exported after linking.

dllimport: May be inlined.  If inlining fails, an import-style call
(__imp_*) is emitted.

Basically, it avoids COMDAT cruft when we can be sure a definition will be
provided by some imported dll.  That feels like a quality of implementation
optimization.  I suppose someone could be using /Ob0 or -fno-inline to
ignore the definition from the header file and always get the import, but
that seems like a corner case.

> The question now is how to implement this. After a brief discussion, I
> can think of four ways:
>
> 1. Add additional linkage type(s) for the combinations to
>    GlobalValue::LinkageTypes.
>
>    This appears to be the least invasive way, but adds new linkage types
>    to an already large list.
>
> 2. Handle dllexport/import similar to ELF visibility by adding new
>    "visibility" types to GlobalValue::VisibilityTypes and IR
visibility
>    styles.
>
>    This feels like kind of a band-aid. While dllexport could be
>    construed as similar to default visibility (some code uses both in
>    the same place depending on platform), dllimport feels wrong here.
>    This would also prevent mixing ELF visibility with dllexport/import.
>
>    The size of GlobalValue can be kept the same by simply adjusting the
>    bit-fields for linkage and visibility.
>
> 3. Add a new enum for dllimport/export to GlobalValue and IR global
>    variables and functions, similar to ELF visibility.
>
>    This is similar to (2.), without the awkward piggybacking on
>    visibility. But it requires extensions to IR just for a single
>    platform.
>
>    The size of GlobalValue can be kept the same by simply adjusting the
>    bit-fields.
>
> 4. Handle dllexport/import as platform-specific IR function attributes.
>
>    Because dllexport/import can also apply to globals which have no
>    attributes, this requires keeping the dllexport/import linkage types
>    just for them.
>
>    Inline does not apply to globals, but MSVC can actually produce
>    initialized dllexported globals, placed in COMDAT sections, with
>    __declspec(selectany). I have no idea if anyone actually does this.
>    LLVM also does not support __declspec(selectany) yet.
>
> There may be even more or better ways to implement this. It may be good
> to keep a future implementation of __declspec(selectany) in mind when
> thinking about this issue.
>
I'm not super familiar with this code, but I think approach 1 is most
consistent with the existing code.  There are already a handful of linkage
types that represent combinations of properties (weak, odr, external,
internal, etc).  It kind of limits the number of linkage combinations that
LLVM needs to think about or support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130326/39c2ead1/attachment.html>

Hi Nico,

We already discussed this on IRC, I just wanted to mention something explicitly.
> 1. Add additional linkage type(s) for the combinations to
>    GlobalValue::LinkageTypes.
>
>    This appears to be the least invasive way, but adds new linkage types
>    to an already large list.
It also will require modifying all the existing code wrt linkonce
semantics of "dllexpport_odr" and "dllimport_odr" stuff.
Which might
be nontrivial (it's hard to say anything in advance).

After all dllexport is basically the same as external linkage and
dllimport - as  well (but just decl). So, instead of adding new
entities here we'd think about the overall picture once again and make
dllexport / dllimport as some sort of flags?

Target-specific attributes seems the best solution here, but they
cannot be applied to globals. Maybe we'd think into this direction?

--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

On 27.03.2013 21:39, Anton Korobeynikov wrote:
> Target-specific attributes seems the best solution here, but they
> cannot be applied to globals. Maybe we'd think into this direction?
I agree. I actually was thinking of this earlier. Globals already have a
comma-separated list for additional properties (namely section and
alignment), but nothing easily extensible like function attributes. This
could be extended for dllexport/import.
This begs the question whether Globals should have proper attributes
like functions. Are there other things this might be beneficial for?

-Nico

Hi Nico, Reid, and Anton,

I missed the discussion when I implemented dllexport/dllimport for our local
tree. I
essentially implemented your approach#1. I was trying to avoid the various
external_linkage + some_attribute approaches because it seems that
external_linkage
would imply the external linkage without the dllimport/dllexport semantics, and
there
may be existing compiler codes that rely on this semantics. If I used
external_linkage
+ some attribute, then any place in existing codes that uses
hasExternalLInkage() will
have to be modified to check both the linkage type as well as the attribute
list, which
seems a bit fragile to maintain; somebody somewhere will forget to check the
attributes
when writing their new optimization or garbage collection passes.

I think dllimport inline functions are more closely related to the
available_externally
linkage than linkonce_odr; no COMDAT is involved and no .linkonce discard
directive
is needed. I would name it dllimport_inline or something similar.

While it makes sense to mimic MSVC behavior and MSDN doc for C++ codes, it is
not very clear to me what should be the correct semantics when a C99 inline
function
carries a dllimport/dllexport declspec. For example,

inline __declspec(dllimport) int foo() { return 100; }
extern inline int foo();

Currently clang ignores the dllimport attribute, so what's left will be:
inline int foo() { return 100; }
extern inline int foo(); // this declaration demands an out-of-line definition

The second declaration with the "extern" keyword demands that foo()
have an out-
of-line function definition. If foo() is actually being dllexported from a DLL,
there will be
duplicate definitions at load time (I think some calls will resolve to one
symbol and others
will resolve to a different symbol foo).

If clang wants to respect the dllimport attribute, then the two declarations
would have a
conflict regarding whether a function body should be generated/externally
visible.

An alternative is to generate an error message whenever a C99 inline function is
used
with a dllimport/dllexport declspec.

In a similar manner, I wonder what should be the correct semantics of:
inline __declspec(dllexport) int foo() { return 100; }

I guess I could let dllexport imply "extern", so that an out-of-line
definition always gets
generated, but is that the expected behavior?

I built a mingw compiler from this following revision and asked it to compile
three C files.
i386-mingw32-gcc (GCC) 4.9.0 20130416 (experimental)

For the following test case, GCC ignores the dllimport attribute with a warning
(similar
to clang's current behavior)

/* test1.c */
inline __declspec(dllimport) void foo();
inline void foo();
inline void foo() { }
void foo_user() { foo(); }
/* end of file */

But for the following two cases, GCC produces both an out-of-line definition and
also a
reference to an dllimported symbol foo with no diagnostics. It seems wrong,
unless
there is some problem with my build process.

/* test2.c */
inline void foo();
__declspec(dllimport) void foo();
inline void foo() { }
void foo_user() { foo(); }
/* end of file */

/* test3.c */
inline void foo();
__declspec(dllimport) void foo();
__declspec(dllimport) void foo() { } // clang complains that this is a
redeclaration without
                                                        // "dllimport"
attribute", which seems wrong...
void foo_user() { foo(); }
/* end of file */

What are your thoughts?
- Gao.

________________________________________
From: llvmdev-bounces at cs.uiuc.edu [llvmdev-bounces at cs.uiuc.edu] on behalf
of Nico Rieck [nico.rieck at gmail.com]
Sent: Monday, March 25, 2013 9:43 PM
To: LLVM Developers Mailing List
Subject: [LLVMdev] Feedback required on proper dllexport/import implementation

Hello,

while improving and extending support for dllexport/import I have
noticed that the current way these are implemented is problematic and I
would like some input on how to proceed.

Currently dllexport/dllimport is treated as linkage type. This conflicts
with inlined functions because there is no linkage for the combination
of both. On first though, combining both doesn't make sense, but take
this class as an example:

  struct __declspec(dllexport/dllimport) X {
    X() {}
  };

Such constructs with empty or simple inline functions can be found for
example in Microsoft headers or even libc++ headers.

Ignoring the dllexport/import attribute for such functions would seem
most sensible (and can be implemented easily), but MSVC does the
opposite: For imported inline functions the function body is dropped and
__imp_ calls are emitted, and exported inline functions are placed into
COMDAT sections. The latter cannot be expressed because it requires
linkonce_odr and dllexport linkage.

The question now is how to implement this. After a brief discussion, I
can think of four ways:

1. Add additional linkage type(s) for the combinations to
   GlobalValue::LinkageTypes.

   This appears to be the least invasive way, but adds new linkage types
   to an already large list.

2. Handle dllexport/import similar to ELF visibility by adding new
   "visibility" types to GlobalValue::VisibilityTypes and IR
visibility
   styles.

   This feels like kind of a band-aid. While dllexport could be
   construed as similar to default visibility (some code uses both in
   the same place depending on platform), dllimport feels wrong here.
   This would also prevent mixing ELF visibility with dllexport/import.

   The size of GlobalValue can be kept the same by simply adjusting the
   bit-fields for linkage and visibility.

3. Add a new enum for dllimport/export to GlobalValue and IR global
   variables and functions, similar to ELF visibility.

   This is similar to (2.), without the awkward piggybacking on
   visibility. But it requires extensions to IR just for a single
   platform.

   The size of GlobalValue can be kept the same by simply adjusting the
   bit-fields.

4. Handle dllexport/import as platform-specific IR function attributes.

   Because dllexport/import can also apply to globals which have no
   attributes, this requires keeping the dllexport/import linkage types
   just for them.

   Inline does not apply to globals, but MSVC can actually produce
   initialized dllexported globals, placed in COMDAT sections, with
   __declspec(selectany). I have no idea if anyone actually does this.
   LLVM also does not support __declspec(selectany) yet.

There may be even more or better ways to implement this. It may be good
to keep a future implementation of __declspec(selectany) in mind when
thinking about this issue.


-Nico

_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Tue, Apr 23, 2013 at 1:10 PM, Gao, Yunzhong <Yunzhong.Gao at
am.sony.com> wrote:
> Hi Nico, Reid, and Anton,
>
> I missed the discussion when I implemented dllexport/dllimport for our
local tree. I
> essentially implemented your approach#1. I was trying to avoid the various
> external_linkage + some_attribute approaches because it seems that
external_linkage
> would imply the external linkage without the dllimport/dllexport semantics,
and there
> may be existing compiler codes that rely on this semantics. If I used
external_linkage
> + some attribute, then any place in existing codes that uses
hasExternalLInkage() will
> have to be modified to check both the linkage type as well as the attribute
list, which
> seems a bit fragile to maintain; somebody somewhere will forget to check
the attributes
> when writing their new optimization or garbage collection passes.
>
> I think dllimport inline functions are more closely related to the
available_externally
> linkage than linkonce_odr; no COMDAT is involved and no .linkonce discard
directive
> is needed. I would name it dllimport_inline or something similar.
Ah, you're right.  dllimport_inline is basically available_externally,
with an indirect call through [__imp_foo] if we choose not to inline.
> While it makes sense to mimic MSVC behavior and MSDN doc for C++ codes, it
is
> not very clear to me what should be the correct semantics when a C99 inline
function
> carries a dllimport/dllexport declspec. For example,
>
> inline __declspec(dllimport) int foo() { return 100; }
> extern inline int foo();
>
> Currently clang ignores the dllimport attribute, so what's left will
be:
> inline int foo() { return 100; }
> extern inline int foo(); // this declaration demands an out-of-line
definition
>
> The second declaration with the "extern" keyword demands that
foo() have an out-
> of-line function definition. If foo() is actually being dllexported from a
DLL, there will be
> duplicate definitions at load time (I think some calls will resolve to one
symbol and others
> will resolve to a different symbol foo).
>
> If clang wants to respect the dllimport attribute, then the two
declarations would have a
> conflict regarding whether a function body should be generated/externally
visible.
>
> An alternative is to generate an error message whenever a C99 inline
function is used
> with a dllimport/dllexport declspec.
Yep, diagnosing this sounds best.  MSVC doesn't support C99 inline so
there shouldn't be much of this kind of code out there.
> In a similar manner, I wonder what should be the correct semantics of:
> inline __declspec(dllexport) int foo() { return 100; }
>
> I guess I could let dllexport imply "extern", so that an
out-of-line definition always gets
> generated, but is that the expected behavior?
Yes, I believe dllexport requires that we instantiate a foo that gets
linked once, and is also exported, like C99 extern inline.
> I built a mingw compiler from this following revision and asked it to
compile three C files.
> i386-mingw32-gcc (GCC) 4.9.0 20130416 (experimental)
>
> For the following test case, GCC ignores the dllimport attribute with a
warning (similar
> to clang's current behavior)
>
> /* test1.c */
> inline __declspec(dllimport) void foo();
> inline void foo();
> inline void foo() { }
> void foo_user() { foo(); }
> /* end of file */
>
> But for the following two cases, GCC produces both an out-of-line
definition and also a
> reference to an dllimported symbol foo with no diagnostics. It seems wrong,
unless
> there is some problem with my build process.
>
> /* test2.c */
> inline void foo();
> __declspec(dllimport) void foo();
> inline void foo() { }
> void foo_user() { foo(); }
> /* end of file */
>
> /* test3.c */
> inline void foo();
> __declspec(dllimport) void foo();
> __declspec(dllimport) void foo() { } // clang complains that this is a
redeclaration without
>                                                         //
"dllimport" attribute", which seems wrong...
> void foo_user() { foo(); }
> /* end of file */
>
> What are your thoughts?
> - Gao.
>
> ________________________________________
> From: llvmdev-bounces at cs.uiuc.edu [llvmdev-bounces at cs.uiuc.edu] on
behalf of Nico Rieck [nico.rieck at gmail.com]
> Sent: Monday, March 25, 2013 9:43 PM
> To: LLVM Developers Mailing List
> Subject: [LLVMdev] Feedback required on proper dllexport/import
implementation
>
> Hello,
>
> while improving and extending support for dllexport/import I have
> noticed that the current way these are implemented is problematic and I
> would like some input on how to proceed.
>
> Currently dllexport/dllimport is treated as linkage type. This conflicts
> with inlined functions because there is no linkage for the combination
> of both. On first though, combining both doesn't make sense, but take
> this class as an example:
>
>   struct __declspec(dllexport/dllimport) X {
>     X() {}
>   };
>
> Such constructs with empty or simple inline functions can be found for
> example in Microsoft headers or even libc++ headers.
>
> Ignoring the dllexport/import attribute for such functions would seem
> most sensible (and can be implemented easily), but MSVC does the
> opposite: For imported inline functions the function body is dropped and
> __imp_ calls are emitted, and exported inline functions are placed into
> COMDAT sections. The latter cannot be expressed because it requires
> linkonce_odr and dllexport linkage.
>
> The question now is how to implement this. After a brief discussion, I
> can think of four ways:
>
> 1. Add additional linkage type(s) for the combinations to
>    GlobalValue::LinkageTypes.
>
>    This appears to be the least invasive way, but adds new linkage types
>    to an already large list.
>
> 2. Handle dllexport/import similar to ELF visibility by adding new
>    "visibility" types to GlobalValue::VisibilityTypes and IR
visibility
>    styles.
>
>    This feels like kind of a band-aid. While dllexport could be
>    construed as similar to default visibility (some code uses both in
>    the same place depending on platform), dllimport feels wrong here.
>    This would also prevent mixing ELF visibility with dllexport/import.
>
>    The size of GlobalValue can be kept the same by simply adjusting the
>    bit-fields for linkage and visibility.
>
> 3. Add a new enum for dllimport/export to GlobalValue and IR global
>    variables and functions, similar to ELF visibility.
>
>    This is similar to (2.), without the awkward piggybacking on
>    visibility. But it requires extensions to IR just for a single
>    platform.
>
>    The size of GlobalValue can be kept the same by simply adjusting the
>    bit-fields.
>
> 4. Handle dllexport/import as platform-specific IR function attributes.
>
>    Because dllexport/import can also apply to globals which have no
>    attributes, this requires keeping the dllexport/import linkage types
>    just for them.
>
>    Inline does not apply to globals, but MSVC can actually produce
>    initialized dllexported globals, placed in COMDAT sections, with
>    __declspec(selectany). I have no idea if anyone actually does this.
>    LLVM also does not support __declspec(selectany) yet.
>
> There may be even more or better ways to implement this. It may be good
> to keep a future implementation of __declspec(selectany) in mind when
> thinking about this issue.
>
>
> -Nico
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>

On 23.04.2013 19:10, Gao, Yunzhong wrote:
> I missed the discussion when I implemented dllexport/dllimport for our
local tree. I
> essentially implemented your approach#1. I was trying to avoid the various
> external_linkage + some_attribute approaches because it seems that
external_linkage
> would imply the external linkage without the dllimport/dllexport semantics,
and there
> may be existing compiler codes that rely on this semantics. If I used
external_linkage
> + some attribute, then any place in existing codes that uses
hasExternalLInkage() will
> have to be modified to check both the linkage type as well as the attribute
list, which
> seems a bit fragile to maintain; somebody somewhere will forget to check
the attributes
> when writing their new optimization or garbage collection passes.
What problems do you expect with another approach? I have a local 
prototype that removes dllimport/export as linkage and uses function 
attributes and an extension to globals. I decorate dllexported functions 
with the Used attribute and it seems to work fine. This allows such 
functions to have linkonce_odr and weak_odr linkage and not be discarded.
> I think dllimport inline functions are more closely related to the
available_externally
> linkage than linkonce_odr; no COMDAT is involved and no .linkonce discard
directive
> is needed. I would name it dllimport_inline or something similar.
Hm, I don't think so, because dllimported inline functions must never be 
instantiated, or else you get duplicated symbols. They are only allowed 
to be expanded. At the moment I solve this by checking for definitions 
during codegen (this means they survived the inliner) and dropping their 
body.
> While it makes sense to mimic MSVC behavior and MSDN doc for C++ codes, it
is
> not very clear to me what should be the correct semantics when a C99 inline
function
> carries a dllimport/dllexport declspec. [...]
> An alternative is to generate an error message whenever a C99 inline
function is used
> with a dllimport/dllexport declspec.
I agree. This should be rejected.
> In a similar manner, I wonder what should be the correct semantics of:
> inline __declspec(dllexport) int foo() { return 100; }
>
> I guess I could let dllexport imply "extern", so that an
out-of-line definition always gets
> generated, but is that the expected behavior?
Yes, such a function is always instanciated and exported.
> For the following test case, GCC ignores the dllimport attribute with a
warning (similar
> to clang's current behavior)
>
> /* test1.c */
> inline __declspec(dllimport) void foo();
> inline void foo();
> inline void foo() { }
> void foo_user() { foo(); }
> /* end of file */
Generally, MSVC allows a dllimport definition if the first declaration 
is specified as inline. This test case should import foo() (or expand it 
inline).
> But for the following two cases, GCC produces both an out-of-line
definition and also a
> reference to an dllimported symbol foo with no diagnostics. It seems wrong,
unless
> there is some problem with my build process.
>
> /* test2.c */
> inline void foo();
> __declspec(dllimport) void foo();
> inline void foo() { }
> void foo_user() { foo(); }
> /* end of file */
Having both seems wrong, as it leads to linker errors. MSVC rejects this 
code because it does not allow redeclaring a function and adding dllimport.
> /* test3.c */
> inline void foo();
> __declspec(dllimport) void foo();
> __declspec(dllimport) void foo() { } // clang complains that this is a
redeclaration without
>                                                          //
"dllimport" attribute", which seems wrong...
> void foo_user() { foo(); }
> /* end of file */
Same as above.
> What are your thoughts?
I have one issue in mind where I wonder if it's beneficial to be less 
strict than MSVC. As stated before, MSVC requires the inline keyword 
together with dllimport. This extends to member functions and prevents 
out-of-line definitions with inline:

   struct X1 {
     inline void foo();
     void bar();
   };
   void X::foo() {} // OK
   inline void X::bar() {} // Error

Same goes for class and function templates. I stumbled on this in libc++ 
where internal class templates are externed.


-Nico