llvm dev - Jun 2015 - [LLVMdev] Linking modules across contexts crashes

I get a crash when I try to link multiple modules registered in their 
individual contexts.
Documentation for Linker::LinkModules doesn't mention anything about 
contexts, and the first link succeeds. But the second link crashes.

Is this not the right way to merge such modules? If not, then what is 
the right way?

In any case, documentation for Linker::LinkModules should say if 
contexts are or aren't expected to be the same.

Yuri


---testcase---
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Module.h"
#include "llvm/Linker/Linker.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/IR/DiagnosticPrinter.h"
#include <string>
#include <iostream>

using namespace llvm;
using namespace std;

int main() {
   // vars
   string Message;
   raw_string_ostream Stream(Message);
   DiagnosticPrinterRawOStream DP(Stream);
   LLVMBool Result;
   // create blank modules and contexts
   LLVMContext *ctx1 = new LLVMContext;
   Module* module1 = new Module("module1", *ctx1);
   LLVMContext *ctx2 = new LLVMContext;
   Module* module2 = new Module("module2", *ctx2);
   LLVMContext *ctx3 = new LLVMContext;
   Module* module3 = new Module("module3", *ctx3);
   // fill modules
llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx1), 
false), llvm::Function::ExternalLinkage, "f1", module1);
llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx2), 
false), llvm::Function::ExternalLinkage, "f2", module2);
llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx3), 
false), llvm::Function::ExternalLinkage, "f3", module3);

   // merge f1 <- f2
   Result = Linker::LinkModules(module1, module2, [&](const 
DiagnosticInfo &DI) {DI.print(DP);});
   cout << "merge result=" << Result << endl;
   delete ctx2;
   cout << "--done merge #1--" << endl;

   // merge f3 <- f1
   Result = Linker::LinkModules(module3, module1, [&](const 
DiagnosticInfo &DI) {DI.print(DP);});
   cout << "merge result=" << Result << endl;
   delete ctx1;
   cout << "--done merge #2--" << endl;

   return 0;
}

---output---
merge result=0
--done merge #1--
Bus error

rev.237344

I'm pretty sure module linking is expected to occur in the same LLVM
context. IIRC Duncan had some proposal for how ld64 could do something
clever with multiple contexts, but I've totally forgotten what it was.

On Fri, May 29, 2015 at 5:18 PM, Yuri <yuri at rawbw.com> wrote:
> I get a crash when I try to link multiple modules registered in their
> individual contexts.
> Documentation for Linker::LinkModules doesn't mention anything about
> contexts, and the first link succeeds. But the second link crashes.
>
> Is this not the right way to merge such modules? If not, then what is the
> right way?
>
> In any case, documentation for Linker::LinkModules should say if contexts
> are or aren't expected to be the same.
>
> Yuri
>
>
> ---testcase---
> #include "llvm/IR/LLVMContext.h"
> #include "llvm/IR/Module.h"
> #include "llvm/Linker/Linker.h"
> #include "llvm/Support/raw_ostream.h"
> #include "llvm/IR/DiagnosticPrinter.h"
> #include <string>
> #include <iostream>
>
> using namespace llvm;
> using namespace std;
>
> int main() {
>   // vars
>   string Message;
>   raw_string_ostream Stream(Message);
>   DiagnosticPrinterRawOStream DP(Stream);
>   LLVMBool Result;
>   // create blank modules and contexts
>   LLVMContext *ctx1 = new LLVMContext;
>   Module* module1 = new Module("module1", *ctx1);
>   LLVMContext *ctx2 = new LLVMContext;
>   Module* module2 = new Module("module2", *ctx2);
>   LLVMContext *ctx3 = new LLVMContext;
>   Module* module3 = new Module("module3", *ctx3);
>   // fill modules
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx1),
> false), llvm::Function::ExternalLinkage, "f1", module1);
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx2),
> false), llvm::Function::ExternalLinkage, "f2", module2);
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx3),
> false), llvm::Function::ExternalLinkage, "f3", module3);
>
>   // merge f1 <- f2
>   Result = Linker::LinkModules(module1, module2, [&](const
DiagnosticInfo
> &DI) {DI.print(DP);});
>   cout << "merge result=" << Result << endl;
>   delete ctx2;
>   cout << "--done merge #1--" << endl;
>
>   // merge f3 <- f1
>   Result = Linker::LinkModules(module3, module1, [&](const
DiagnosticInfo
> &DI) {DI.print(DP);});
>   cout << "merge result=" << Result << endl;
>   delete ctx1;
>   cout << "--done merge #2--" << endl;
>
>   return 0;
> }
>
> ---output---
> merge result=0
> --done merge #1--
> Bus error
>
> rev.237344
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150601/dbb247c6/attachment.html>

> On 2015-Jun-01, at 11:06, Reid Kleckner <rnk at google.com> wrote:
> 
> I'm pretty sure module linking is expected to occur in the same LLVM
context.
Correct.
> IIRC Duncan had some proposal for how ld64 could do something clever with
multiple contexts, but I've totally forgotten what it was.
This was for LTO (probably unrelated to Yuri's scenario?).

 1. Lazy-load each module in its own context, parse its symbols
    (without actually loading functions, metadata, etc.), and
    destroy the module and context.
 2. Decide which modules we should link (based on symbol parsing).
 3. Load all the chosen modules into a single context and link them
    together.

lib/LTO has a bunch of the pieces for this, but I got the idea from
Rafael's work on tools/gold-plugin/gold-plugin.cpp.  It might be
easier to look at there.
> On Fri, May 29, 2015 at 5:18 PM, Yuri <yuri at rawbw.com> wrote:
> I get a crash when I try to link multiple modules registered in their
individual contexts.
> Documentation for Linker::LinkModules doesn't mention anything about
contexts, and the first link succeeds. But the second link crashes.
The problem is that a fair number of things referenced by a Module
are owned by its LLVMContext (types, constants, metadata, and some
other things).
> Is this not the right way to merge such modules? If not, then what is the
right way?
You can round-trip to bitcode, reading the module into the
destination context.  The following pseudo-code gives the idea:

    bool linkModuleFromDifferentContext(Module &D, const Module &S) {
      SmallVector<char, 256> Buffer;
      writeBitcodeToBuffer(S, Buffer);

      std::unique_ptr<Module> M = readBitcodeFromBuffer(D.getContext());
      return Linker::LinkModules(&D, M.get());
    }
> In any case, documentation for Linker::LinkModules should say if contexts
are or aren't expected to be the same.
Good idea; patch welcome.
> 
> Yuri
> 
> 
> ---testcase---
> #include "llvm/IR/LLVMContext.h"
> #include "llvm/IR/Module.h"
> #include "llvm/Linker/Linker.h"
> #include "llvm/Support/raw_ostream.h"
> #include "llvm/IR/DiagnosticPrinter.h"
> #include <string>
> #include <iostream>
> 
> using namespace llvm;
> using namespace std;
> 
> int main() {
>   // vars
>   string Message;
>   raw_string_ostream Stream(Message);
>   DiagnosticPrinterRawOStream DP(Stream);
>   LLVMBool Result;
>   // create blank modules and contexts
>   LLVMContext *ctx1 = new LLVMContext;
>   Module* module1 = new Module("module1", *ctx1);
>   LLVMContext *ctx2 = new LLVMContext;
>   Module* module2 = new Module("module2", *ctx2);
>   LLVMContext *ctx3 = new LLVMContext;
>   Module* module3 = new Module("module3", *ctx3);
>   // fill modules
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx1),
false), llvm::Function::ExternalLinkage, "f1", module1);
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx2),
false), llvm::Function::ExternalLinkage, "f2", module2);
> llvm::Function::Create(llvm::FunctionType::get(Type::getInt32Ty(*ctx3),
false), llvm::Function::ExternalLinkage, "f3", module3);
> 
>   // merge f1 <- f2
>   Result = Linker::LinkModules(module1, module2, [&](const
DiagnosticInfo &DI) {DI.print(DP);});
>   cout << "merge result=" << Result << endl;
>   delete ctx2;
>   cout << "--done merge #1--" << endl;
> 
>   // merge f3 <- f1
>   Result = Linker::LinkModules(module3, module1, [&](const
DiagnosticInfo &DI) {DI.print(DP);});
>   cout << "merge result=" << Result << endl;
>   delete ctx1;
>   cout << "--done merge #2--" << endl;
> 
>   return 0;
> }
> 
> ---output---
> merge result=0
> --done merge #1--
> Bus error
> 
> rev.237344
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>