llvm dev - Aug 2021 - Have LLD load a file through memory

Hi.

I'm trying to use LLD to get it's input file from a string in memory
instead of a file.

My use case is a JIT compiler (IR --> AMD GCN ISA) where calling the linker
to make it a loadable object via system() is hitting a bug (somewhere in the
system).

Here's what I do:

  1.  With the AMDGPU backend I write the output to a string
  2.  I sneak in this string into LLD via an added access function.
  3.  Since LLD invocation is only used in this form:
ld.lld -shared k.o -o k.so
I modified 'elf::readFile' to

  std::string tmp = qdp_jit_get_object();   // this retrieves the sneaky string

  llvm::outs() << "size   = " << tmp.size() <<
"\n";

  std::unique_ptr<MemoryBuffer> mb = MemoryBuffer::getMemBuffer(
StringRef( tmp.c_str() , tmp.length() ) );

  MemoryBufferRef mbref = mb->getMemBufferRef();

  return mbref;


When I then call LLD via lld::elf::link, I get

size   = 2192
ld: error: :1: unknown directive:
>>> A�E�A�E�@���?V@���?V�3��������}j<������� �J�0�ւ���j�
>>> ^
This is how the string containing the object was created:

   string str_obj;
   llvm::raw_string_ostream rss(str_obj);
   llvm::buffer_ostream bos(rss);


   if (TargetMachine->addPassesToEmitFile(CodeGenPasses,
  bos,
  nullptr,
  llvm::CodeGenFileType::CGFT_ObjectFile ))

CodeGenPasses.run(*Mod);


Is there anything wrong how I turn the string into a MemoryBuffer so that the
link fails?

Thanks,
Frank


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210812/94c4ae50/attachment.html>

On 2021-08-12, Frank Winter via llvm-dev wrote:
>Hi.
>
>I'm trying to use LLD to get it's input file from a string in memory
instead of a file.
>
>My use case is a JIT compiler (IR --> AMD GCN ISA) where calling the
linker to make it a loadable object via system() is hitting a bug (somewhere in
the system).
>
>Here's what I do:
>
>  1.  With the AMDGPU backend I write the output to a string
>  2.  I sneak in this string into LLD via an added access function.
>  3.  Since LLD invocation is only used in this form:
>ld.lld -shared k.o -o k.so
>I modified 'elf::readFile' to
>
>  std::string tmp = qdp_jit_get_object();   // this retrieves the sneaky
string
>
>  llvm::outs() << "size   = " << tmp.size() <<
"\n";
>
>  std::unique_ptr<MemoryBuffer> mb = MemoryBuffer::getMemBuffer(
StringRef( tmp.c_str() , tmp.length() ) );
>
>  MemoryBufferRef mbref = mb->getMemBufferRef();
>
>  return mbref;
>
>
>When I then call LLD via lld::elf::link, I get
>
>size   = 2192
>ld: error: :1: unknown directive:
>>>> A�E�A�E�@���?V@���?V�3��������}j<������� �J�0�ւ���j�
>>>> ^
Looks like the file you passed to lld/ELF is not an ELF file.
lld/ELF parsed it as a linker script and bailed out.

Note, llvm-project/lld isn't ready when used as a library.
https://lists.llvm.org/pipermail/llvm-dev/2021-June/151031.html ("RFC:
Revisiting LLD-as-a-library design") lists some issues.

* cl::ResetAllOptionOccurrences && ParseCommandLineOptions() are called.
The cl::opt, cl::list, ... options are reset.
   If the other part of the process using LLVM as a library needs to call into
LLVM, resetting options may be a problem.
* One fatal error or more than --error-limit (default: 20) errors will exit the
process.
   With CrashRecoveryContext, the exit calls becomes a signal raise. There can
be random resource leaks.
* Warnings/erros are printed to stderr, which may interfere with the other part
of the process.

Also be wary of the default uses all cores. If you have parallel links, you may
need to specify
--threads=N.

People sometimes find lld::elf::link easy so use it instead of spawning a
process, but there are quite a few caveats.
>This is how the string containing the object was created:
>
>   string str_obj;
>   llvm::raw_string_ostream rss(str_obj);
>   llvm::buffer_ostream bos(rss);
>
>
>   if (TargetMachine->addPassesToEmitFile(CodeGenPasses,
>  bos,
>  nullptr,
>  llvm::CodeGenFileType::CGFT_ObjectFile ))
>
>CodeGenPasses.run(*Mod);
>
>
>Is there anything wrong how I turn the string into a MemoryBuffer so that
the link fails?
>
>Thanks,
>Frank
>
>
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev