llvm dev - Nov 2019 - ORC API C Standard/System Libraries

Hi all,

First of all, I want to say that I really appreciate all the work that's
gone into LLVM, clang and the ORC API. I would be very unlikely to be doing
my current project if they weren't available.

I'm using the ORC API to JIT some C code (soon to be C++) on Windows.
(I was on LLVM 8.0.0 but it didn't handle the 'allocator' attribute
so I
upgraded to LLVM 9.0.0.)

The JIT compiling works well for simple files, #included local files and
object files.
As soon as I start trying to use system libraries with functions, things
start to fall down.
I think this is largely just due to my ignorance of clang in general, but
any guidance would be greatly appreciated.


1) I get multiple warnings for the system headers (e.g. stdlib) of the form:
"warning: macro expansion producing 'defined' has undefined
behavior"

Is there a way to prevent these warnings? Preferably only for the system
libraries.
I tried -Wno-expansion-to-defined, -fms-extensions, -fms-compatibility-fms,
-compatibility-version=14.16.27023. I also tried to reference the system
library folder by -isystem, -internal-isystem and -c-isystem


2) Disregarding the warnings, I also get the following error message:
"JIT session error: Symbols not found: { malloc }"

I assumed this might be because I wasn't linking the standard library, so I
tried appending "msvcrt.lib" and "-stdlib=msvcrt.lib" but to
no avail. Can
you use .lib files with clang on Windows?

How do I make sure that the symbols defined in the standard libraries are
available to my JIT?


Thanks for any help you can give!
All the best,
Andrew
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191112/f322d4fd/attachment.html>

+Lang Hames <lhames at gmail.com>, Liberator of the Orcs

On Tue, Nov 12, 2019 at 12:48 PM Andrew Reece via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all,
>
> First of all, I want to say that I really appreciate all the work
that's
> gone into LLVM, clang and the ORC API. I would be very unlikely to be doing
> my current project if they weren't available.
>
> I'm using the ORC API to JIT some C code (soon to be C++) on Windows.
> (I was on LLVM 8.0.0 but it didn't handle the 'allocator'
attribute so I
> upgraded to LLVM 9.0.0.)
>
> The JIT compiling works well for simple files, #included local files and
> object files.
> As soon as I start trying to use system libraries with functions, things
> start to fall down.
> I think this is largely just due to my ignorance of clang in general, but
> any guidance would be greatly appreciated.
>
>
> 1) I get multiple warnings for the system headers (e.g. stdlib) of the
> form:
> "warning: macro expansion producing 'defined' has undefined
behavior"
>
> Is there a way to prevent these warnings? Preferably only for the system
> libraries.
> I tried -Wno-expansion-to-defined, -fms-extensions,
> -fms-compatibility-fms, -compatibility-version=14.16.27023. I also tried to
> reference the system library folder by -isystem, -internal-isystem and
> -c-isystem
>
>
> 2) Disregarding the warnings, I also get the following error message:
> "JIT session error: Symbols not found: { malloc }"
>
> I assumed this might be because I wasn't linking the standard library,
so
> I tried appending "msvcrt.lib" and "-stdlib=msvcrt.lib"
but to no avail.
> Can you use .lib files with clang on Windows?
>
> How do I make sure that the symbols defined in the standard libraries are
> available to my JIT?
>
>
> Thanks for any help you can give!
> All the best,
> Andrew
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191118/32936f23/attachment.html>

Hi Andrew,

1) I get multiple warnings for the system headers (e.g. stdlib) of the
form:
> "warning: macro expansion producing 'defined' has undefined
behavior"
> Is there a way to prevent these warnings? Preferably only for the system
> libraries.
> I tried -Wno-expansion-to-defined, -fms-extensions,
> -fms-compatibility-fms, -compatibility-version=14.16.27023. I also tried to
> reference the system library folder by -isystem, -internal-isystem and
> -c-isystem

I'm afraid I don't have any insight on this. If nobody chimes in on this
list it may be worth asking this on the clang-dev mailing list.

2) Disregarding the warnings, I also get the following error
message:
> "JIT session error: Symbols not found: { malloc }"

How is your JIT set up? Are you using LLJIT, or a custom JIT class?

Either way, the key is to make sure that process symbols are added to a
JITDylib so that they can be found by JIT'd code. As an example, the code
to do this in LLVM's lli tool is:

J->getMainJITDylib().addGenerator(
      ExitOnErr(orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
          J->getDataLayout().getGlobalPrefix())));

Cheers,
Lang.


On Tue, Nov 12, 2019 at 12:48 PM Andrew Reece via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi all,
>
> First of all, I want to say that I really appreciate all the work
that's
> gone into LLVM, clang and the ORC API. I would be very unlikely to be doing
> my current project if they weren't available.
>
> I'm using the ORC API to JIT some C code (soon to be C++) on Windows.
> (I was on LLVM 8.0.0 but it didn't handle the 'allocator'
attribute so I
> upgraded to LLVM 9.0.0.)
>
> The JIT compiling works well for simple files, #included local files and
> object files.
> As soon as I start trying to use system libraries with functions, things
> start to fall down.
> I think this is largely just due to my ignorance of clang in general, but
> any guidance would be greatly appreciated.
>
>
> 1) I get multiple warnings for the system headers (e.g. stdlib) of the
> form:
> "warning: macro expansion producing 'defined' has undefined
behavior"
>
> Is there a way to prevent these warnings? Preferably only for the system
> libraries.
> I tried -Wno-expansion-to-defined, -fms-extensions,
> -fms-compatibility-fms, -compatibility-version=14.16.27023. I also tried to
> reference the system library folder by -isystem, -internal-isystem and
> -c-isystem
>
>
> 2) Disregarding the warnings, I also get the following error message:
> "JIT session error: Symbols not found: { malloc }"
>
> I assumed this might be because I wasn't linking the standard library,
so
> I tried appending "msvcrt.lib" and "-stdlib=msvcrt.lib"
but to no avail.
> Can you use .lib files with clang on Windows?
>
> How do I make sure that the symbols defined in the standard libraries are
> available to my JIT?
>
>
> Thanks for any help you can give!
> All the best,
> Andrew
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191121/80d6f88c/attachment.html>

Hi Lang,

Thanks for the response.

I'm using a custom JIT that was copied from LLJIT and then slightly edited.
It's jitting to memory that is shared with a child process and the child
process runs the 'code'.
I assume this generator won't work as the symbols won't be in the
current
process.

My recollection is that LLVM used to have remote process support but has
since deprecated it. Is that correct?

My working model was that clang would create all the symbols that are used
(both from a user code file and from system library), these would go into
the IR, and then the JIT would turn these into executable code.
This doesn't seem to be correct given that the library symbols aren't
being
found. (Is this because I'm not statically linking the system libs? I'll
check this out.)

I'm not sure whether I'm misunderstanding the clang side, the LLVM side,
something else, or some combination of the above!
I'll have another look at the IR now and see if I can work anything out but
any light you (or anyone else) can shed on the matter would be greatly
appreciated.

All the best,
Andrew

On Thu, 21 Nov 2019 at 19:56, Lang Hames <lhames at gmail.com> wrote:
> Hi Andrew,
>
> 1) I get multiple warnings for the system headers (e.g. stdlib) of the
>> form:
>> "warning: macro expansion producing 'defined' has
undefined behavior"
>> Is there a way to prevent these warnings? Preferably only for the
system
>> libraries.
>> I tried -Wno-expansion-to-defined, -fms-extensions,
>> -fms-compatibility-fms, -compatibility-version=14.16.27023. I also
tried to
>> reference the system library folder by -isystem, -internal-isystem and
>> -c-isystem
>
>
> I'm afraid I don't have any insight on this. If nobody chimes in on
this
> list it may be worth asking this on the clang-dev mailing list.
>
> 2) Disregarding the warnings, I also get the following error message:
>> "JIT session error: Symbols not found: { malloc }"
>
>
> How is your JIT set up? Are you using LLJIT, or a custom JIT class?
>
> Either way, the key is to make sure that process symbols are added to a
> JITDylib so that they can be found by JIT'd code. As an example, the
code
> to do this in LLVM's lli tool is:
>
> J->getMainJITDylib().addGenerator(
>       ExitOnErr(orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
>           J->getDataLayout().getGlobalPrefix())));
>
> Cheers,
> Lang.
>
>
> On Tue, Nov 12, 2019 at 12:48 PM Andrew Reece via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> First of all, I want to say that I really appreciate all the work
that's
>> gone into LLVM, clang and the ORC API. I would be very unlikely to be
doing
>> my current project if they weren't available.
>>
>> I'm using the ORC API to JIT some C code (soon to be C++) on
Windows.
>> (I was on LLVM 8.0.0 but it didn't handle the 'allocator'
attribute so I
>> upgraded to LLVM 9.0.0.)
>>
>> The JIT compiling works well for simple files, #included local files
and
>> object files.
>> As soon as I start trying to use system libraries with functions,
things
>> start to fall down.
>> I think this is largely just due to my ignorance of clang in general,
but
>> any guidance would be greatly appreciated.
>>
>>
>> 1) I get multiple warnings for the system headers (e.g. stdlib) of the
>> form:
>> "warning: macro expansion producing 'defined' has
undefined behavior"
>>
>> Is there a way to prevent these warnings? Preferably only for the
system
>> libraries.
>> I tried -Wno-expansion-to-defined, -fms-extensions,
>> -fms-compatibility-fms, -compatibility-version=14.16.27023. I also
tried to
>> reference the system library folder by -isystem, -internal-isystem and
>> -c-isystem
>>
>>
>> 2) Disregarding the warnings, I also get the following error message:
>> "JIT session error: Symbols not found: { malloc }"
>>
>> I assumed this might be because I wasn't linking the standard
library, so
>> I tried appending "msvcrt.lib" and
"-stdlib=msvcrt.lib" but to no avail.
>> Can you use .lib files with clang on Windows?
>>
>> How do I make sure that the symbols defined in the standard libraries
are
>> available to my JIT?
>>
>>
>> Thanks for any help you can give!
>> All the best,
>> Andrew
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191126/a791e42e/attachment-0001.html>