thr3ads.net - llvm dev - [LLVMdev] load bytecode from string for jiting problem [Mar 2014]

If this information is useful, please help other people find it:
Share via:

Willy WOLFF

2014-Mar-19 21:58 UTC

[LLVMdev] load bytecode from string for jiting problem

all of:
----
                                // cout << "lsr: " << lsr
<< "\n";
                                llvm::MemoryBuffer* mbjit                       
llvm::MemoryBuffer::getMemBufferCopy (sr);
------
string lsr = sr.str();
                                // cout << "lsr: " << lsr
<< "\n";
                                llvm::MemoryBuffer* mbjit                       
llvm::MemoryBuffer::getMemBuffer (lsr);
-------
string lsr = sr.str();
                                // cout << "lsr: " << lsr
<< "\n";
                                llvm::MemoryBuffer* mbjit                       
llvm::MemoryBuffer::getMemBufferCopy (lsr);


have same result as invalid bit code.
The result of valgrind, effectively, i have invalid reads in the
parseBitcodeFile:

==536== Conditional jump or move depends on uninitialised value(s)
==536==    at 0x501FE3: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x501A19: llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536== 
==536== Invalid read of size 8
==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x501A19: llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==536== 
==536== 
==536== Process terminating with default action of signal 11 (SIGSEGV)
==536==  Access not within mapped region at address 0x0
==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x501A19: llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)



--
Willy WOLFF

On 19 Mar 2014, at 22:11, Vikas Bhargava wrote:
> Hi Willy,
> If the disassembly of the module works fine, then there is nothing wrong
with the module.
> 
> Stream uses the memorybuffer that you pass in parseBitcodeFile. If what
Will is saying is true, there is something wrong with your code in
"3:", i.e.:
> 
> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>   LLVMContext& context = getGlobalContext();
>   ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit, context);
>   if (error_code EC = ModuleOrErr.getError())
>   {
>     std::cout << ModuleOrErr.getError().
> message() << "\n";
>     assert(false);
>   }
> 
> Can you post how you modified it in your second reply? For debugging
purpose, you can simply use MemoryBuffer::getMemBufferCopy() and not worry about
validity of stringref or null-termination. Also, you can run your program
through valgrind and check for any invalid reads.
> 
> HTH
> Vikas.
> ======> 
> 
> 
> On Wed, Mar 19, 2014 at 10:32 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr> wrote:
> I mad the change, and still have the problem.
> 
> I investigate more the source code of llvm.
> 
> First, I change isRawBitcode function to print the content of the parameter
like this:
> original:
http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.html#l00081
> 
>   inline bool isRawBitcode(const unsigned char *BufPtr,
>                            const unsigned char *BufEnd) {
>     // These bytes sort of have a hidden message, but it's not in
>     // little-endian this time, and it's a little redundant.
>           errs()<< "isRawBitcode output:\n";
>           for (int i = 0; i < 4; i++)
>                   errs() << BufPtr[i] << "\n";
>           if (BufPtr != BufEnd )
>                 errs() << "BP != BE ok\n";
>           if (BufPtr[0] == 'B')
>                 errs() << "B ok\n";
>           if (BufPtr[1] == 'C')
>                 errs() << "C ok\n";
>           if (BufPtr[2] ==  0xc0)
>                 errs() << "0xc0 ok\n";
>           if (BufPtr[3] ==  0xde)
>                 errs() << "0xde ok\n";
> 
>     return BufPtr != BufEnd &&
>            BufPtr[0] == 'B' &&
>            BufPtr[1] == 'C' &&
>            BufPtr[2] == 0xc0 &&
>            BufPtr[3] == 0xde;
>   }
> 
> 
> Second, I change ParseBitcodeInto as this:
> original:
http://llvm.org/docs/doxygen/html/BitcodeReader_8cpp_source.html#l01971
> ...
>         errs() << "parsebitcodeinto sniff the signature\n";
>         uint32_t bvar = Stream.Read(8);
>                         errs() << "B :" << bvar
<< "\n";
>         if (bvar != 'B') {
>                 errs() << "B :" << bvar <<
"\n";
>                 return Error(InvalidBitcodeSignature);
>         }
> 
>         if (Stream.Read(8) != 'C') {
>                 errs() << "C\n";
>                 return Error(InvalidBitcodeSignature);
>         }
>         if (  Stream.Read(8) != 0xc0 ) {
>                 errs() << "0xc0\n";
>                 return Error(InvalidBitcodeSignature);
>         }
>         if (  Stream.Read(8) != 0xde ) {
>                 errs() << "0xde\n";
>                 return Error(InvalidBitcodeSignature);
>         }
>         // if (Stream.Read(8) != 'B' ||
>         //     Stream.Read(8) != 'C' ||
>         //     Stream.Read(4) != 0x0 ||
>         //     Stream.Read(4) != 0xC ||
>         //     Stream.Read(4) != 0xE ||
>         //     Stream.Read(4) != 0xD
>         //      ) {
> ...
> 
> 
> 
> The output of the code is :
> 
> 
> isRawBitcode output:
> B
> C
> 
> 
> BP != BE ok
> 
> B ok
> C ok
> 0xc0 ok
> 0xde ok
> 
> parsebitcodeinto sniff the signature
> B :37
> B :37
> 
> 
> 
> 
> It's possible that Stream object is not correctly initialized?
> 
> 
> On 03/13/2014 06:37 PM, Will Dietz wrote:
> On Thu, Mar 13, 2014 at 9:02 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr> wrote:
> Hello,
> 
> I having a weird problem while writing a bytecode module to a string,
> and after read/parse it for unsing on a jit.
> 
> I write a pass to export function to module, and put this module inside
> a global variable.
> I use WriteBitcodeToFile for this.
> For debuging, after this write, I try to load the exported module with
> parseBitcodeFile.
> This two step works.
> 
> 
> 
> After, while the compiled program is running, I try to read and parse
> this global variable for jiting the function.
> 
> 1) I read the global variable with
>    StringRef sr (gv, gv_length);
> 
> 2) I manually test this bytecode by
> (inspired by  inline bool isRawBitcode(const unsigned char *BufPtr,
> const unsigned char *BufEnd) at
> http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.html#l00067)
>    if (sr.str()[0] == 'B')
>      std::cout << "B ok\n";
>    if (sr.str()[1] == 'C')
>      std::cout << "C ok\n";
>    if (sr.str()[2] == (char) 0xc0)
>      std::cout << "0xc0 ok\n";
>    if (sr.str()[3] == (char) 0xde)
>      std::cout << "0xde ok\n";
> 
> 3) I try to parse the gv by
>    MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
> 
> Not sure if this is your issue, but should be fixed anyway:
> 
> The std::string created by "sr.str()" ends its lifetime in this
> statement, and MemoryBuffer for efficiency reasons
> avoids copying data it doesn't have to (like StringRef) so will be
> referencing the freed memory.
> 
> To resolve this:
> * Pass MemoryBuffer your StringRef directly
> * Use getMemBufferCopy()
> * Preserve the result of sr.str() into a stack variable and pass that
> to getMemoryBuffer() instead.
> 
> As a final note, check if your bitcode buffer "string" is
> null-terminated or not.  If not, be sure to be careful and
> do things like informing MemoryBuffer that this is the case.
> 
> Hope this helps,
> ~Will
> 
>    LLVMContext& context = getGlobalContext();
>    ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit, context);
>    if (error_code EC = ModuleOrErr.getError())
>    {
>      std::cout << ModuleOrErr.getError().message() <<
"\n";
>      assert(false);
>    }
> 
> 
> 
> 
> This is the execution result:
> B ok
> C ok
> 0xc0 ok
> 0xde ok
> Invalid bitcode signature
> 
> 
> 
> Ok is not working :/
> But why ???
> 
> 
> 
> For debuging, between 2) and 3), I export the readed module and write to
> a file on my hard drive,
> and try llvm-dis, and the dissasembly of the module works.
> 
> Wath's wrong? Any idea for solve this problem?
> 
> Thanks you very much.
> 
> Regards,
> Willy
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140319/eef13914/attachment.html>

Vikas Bhargava

2014-Mar-19 23:37 UTC

head link

[LLVMdev] load bytecode from string for jiting problem

segmentation fault indicates memory corruption and it's hard to tell
without seeing the exact use of the APIs. If possible, please post a
complete program and gdb stack trace from the core file. If there are
multiple threads using the global variables, please let us know.

FWIW, I have some tests to write llvm::Module to bitcode files and read
them back into llvm::Module and they work just fine with 3.4 (never tried
with tip).

thx
vikas.
=======

On Wed, Mar 19, 2014 at 2:58 PM, Willy WOLFF <willy.wolff at
etu.unistra.fr>wrote:
> all of:
> ----
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >
> llvm::MemoryBuffer::getMemBufferCopy (sr);
> ------
> string lsr = sr.str();
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >             
llvm::MemoryBuffer::getMemBuffer
> (lsr);
> -------
> string lsr = sr.str();
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >
> llvm::MemoryBuffer::getMemBufferCopy (lsr);
>
>
> have same result as invalid bit code.
> The result of valgrind, effectively, i have invalid reads in the
> parseBitcodeFile:
>
> ==536== Conditional jump or move depends on uninitialised value(s)
> ==536==    at 0x501FE3: llvm::BitstreamCursor::Read(unsigned int) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536=> ==536== Invalid read of size 8
> ==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==  Address 0x0 is not stack'd, malloc'd or (recently)
free'd
> ==536=> ==536=> ==536== Process terminating with default action of
signal 11 (SIGSEGV)
> ==536==  Access not within mapped region at address 0x0
> ==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
> llvm::LLVMContext&) (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>
>
>
> *--*
> *Willy WOLFF*
>
> On 19 Mar 2014, at 22:11, Vikas Bhargava wrote:
>
> Hi Willy,
> If the disassembly of the module works fine, then there is nothing wrong
> with the module.
>
> Stream uses the memorybuffer that you pass in parseBitcodeFile. If what
> Will is saying is true, there is something wrong with your code in
"3:",
> i.e.:
>
> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>   LLVMContext& context = getGlobalContext();
>   ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit, context);
>   if (error_code EC = ModuleOrErr.getError())
>   {
>     std::cout << ModuleOrErr.getError().
> message() << "\n";
>     assert(false);
>   }
>
> Can you post how you modified it in your second reply? For debugging
> purpose, you can simply use MemoryBuffer::getMemBufferCopy() and not worry
> about validity of stringref or null-termination. Also, you can run your
> program through valgrind and check for any invalid reads.
>
> HTH
> Vikas.
> ======>
>
>
> On Wed, Mar 19, 2014 at 10:32 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr>wrote:
>
>> I mad the change, and still have the problem.
>>
>> I investigate more the source code of llvm.
>>
>> First, I change isRawBitcode function to print the content of the
>> parameter like this:
>> original: http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.
>> html#l00081
>>
>>   inline bool isRawBitcode(const unsigned char *BufPtr,
>>                            const unsigned char *BufEnd) {
>>     // These bytes sort of have a hidden message, but it's not in
>>     // little-endian this time, and it's a little redundant.
>>           errs()<< "isRawBitcode output:\n";
>>           for (int i = 0; i < 4; i++)
>>                   errs() << BufPtr[i] << "\n";
>>           if (BufPtr != BufEnd )
>>                 errs() << "BP != BE ok\n";
>>           if (BufPtr[0] == 'B')
>>                 errs() << "B ok\n";
>>           if (BufPtr[1] == 'C')
>>                 errs() << "C ok\n";
>>           if (BufPtr[2] ==  0xc0)
>>                 errs() << "0xc0 ok\n";
>>           if (BufPtr[3] ==  0xde)
>>                 errs() << "0xde ok\n";
>>
>>     return BufPtr != BufEnd &&
>>            BufPtr[0] == 'B' &&
>>            BufPtr[1] == 'C' &&
>>            BufPtr[2] == 0xc0 &&
>>            BufPtr[3] == 0xde;
>>   }
>>
>>
>> Second, I change ParseBitcodeInto as this:
>> original: http://llvm.org/docs/doxygen/html/BitcodeReader_8cpp_
>> source.html#l01971
>> ...
>>         errs() << "parsebitcodeinto sniff the
signature\n";
>>         uint32_t bvar = Stream.Read(8);
>>                         errs() << "B :" << bvar
<< "\n";
>>         if (bvar != 'B') {
>>                 errs() << "B :" << bvar <<
"\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>
>>         if (Stream.Read(8) != 'C') {
>>                 errs() << "C\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         if (  Stream.Read(8) != 0xc0 ) {
>>                 errs() << "0xc0\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         if (  Stream.Read(8) != 0xde ) {
>>                 errs() << "0xde\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         // if (Stream.Read(8) != 'B' ||
>>         //     Stream.Read(8) != 'C' ||
>>         //     Stream.Read(4) != 0x0 ||
>>         //     Stream.Read(4) != 0xC ||
>>         //     Stream.Read(4) != 0xE ||
>>         //     Stream.Read(4) != 0xD
>>         //      ) {
>> ...
>>
>>
>>
>> The output of the code is :
>>
>>
>> isRawBitcode output:
>> B
>> C
>>
>>
>> BP != BE ok
>>
>> B ok
>> C ok
>> 0xc0 ok
>> 0xde ok
>>
>> parsebitcodeinto sniff the signature
>> B :37
>> B :37
>>
>>
>>
>>
>> It's possible that Stream object is not correctly initialized?
>>
>>
>> On 03/13/2014 06:37 PM, Will Dietz wrote:
>>
>>> On Thu, Mar 13, 2014 at 9:02 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I having a weird problem while writing a bytecode module to a
string,
>>>> and after read/parse it for unsing on a jit.
>>>>
>>>> I write a pass to export function to module, and put this
module inside
>>>> a global variable.
>>>> I use WriteBitcodeToFile for this.
>>>> For debuging, after this write, I try to load the exported
module with
>>>> parseBitcodeFile.
>>>> This two step works.
>>>>
>>>>
>>>>
>>>> After, while the compiled program is running, I try to read and
parse
>>>> this global variable for jiting the function.
>>>>
>>>> 1) I read the global variable with
>>>>    StringRef sr (gv, gv_length);
>>>>
>>>> 2) I manually test this bytecode by
>>>> (inspired by  inline bool isRawBitcode(const unsigned char
*BufPtr,
>>>> const unsigned char *BufEnd) at
>>>>
http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.html#l00067)
>>>>    if (sr.str()[0] == 'B')
>>>>      std::cout << "B ok\n";
>>>>    if (sr.str()[1] == 'C')
>>>>      std::cout << "C ok\n";
>>>>    if (sr.str()[2] == (char) 0xc0)
>>>>      std::cout << "0xc0 ok\n";
>>>>    if (sr.str()[3] == (char) 0xde)
>>>>      std::cout << "0xde ok\n";
>>>>
>>>> 3) I try to parse the gv by
>>>>    MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>>>>
>>>
>>> Not sure if this is your issue, but should be fixed anyway:
>>>
>>> The std::string created by "sr.str()" ends its lifetime
in this
>>> statement, and MemoryBuffer for efficiency reasons
>>> avoids copying data it doesn't have to (like StringRef) so will
be
>>> referencing the freed memory.
>>>
>>> To resolve this:
>>> * Pass MemoryBuffer your StringRef directly
>>> * Use getMemBufferCopy()
>>> * Preserve the result of sr.str() into a stack variable and pass
that
>>> to getMemoryBuffer() instead.
>>>
>>> As a final note, check if your bitcode buffer "string" is
>>> null-terminated or not.  If not, be sure to be careful and
>>> do things like informing MemoryBuffer that this is the case.
>>>
>>> Hope this helps,
>>> ~Will
>>>
>>>     LLVMContext& context = getGlobalContext();
>>>>    ErrorOr<Module*> ModuleOrErr = parseBitcodeFile
(mbjit, context);
>>>>    if (error_code EC = ModuleOrErr.getError())
>>>>    {
>>>>      std::cout << ModuleOrErr.getError().message()
<< "\n";
>>>>      assert(false);
>>>>    }
>>>>
>>>>
>>>>
>>>>
>>>> This is the execution result:
>>>> B ok
>>>> C ok
>>>> 0xc0 ok
>>>> 0xde ok
>>>> Invalid bitcode signature
>>>>
>>>>
>>>>
>>>> Ok is not working :/
>>>> But why ???
>>>>
>>>>
>>>>
>>>> For debuging, between 2) and 3), I export the readed module and
write to
>>>> a file on my hard drive,
>>>> and try llvm-dis, and the dissasembly of the module works.
>>>>
>>>> Wath's wrong? Any idea for solve this problem?
>>>>
>>>> Thanks you very much.
>>>>
>>>> Regards,
>>>> Willy
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140319/f5cc5ca0/attachment.html>

Willy WOLFF

2014-Mar-20 09:10 UTC

head link

[LLVMdev] load bytecode from string for jiting problem

This segfault occuring only under valgrind,
in shell way, and in gdb way i have 

Invalid bitcode signature
simple_scev_dynamic_array:
/home/willy/apollo/llvm/include/llvm/Support/ErrorOr.h:258: storage_type
*llvm::ErrorOr<llvm::Module *>::getStorage() [T = llvm::Module *]:
Assertion `!HasError && "Cannot get value when an error
exists!"' failed.
Command terminated by signal 6


this is the code I use:


long jitter(void* info, skeleton_pair *skeletons, long skeleton_size, param_t
params, long phi_state_size) {
	InitializeNativeTarget();
	InitializeNativeTargetAsmPrinter();

	llvm::StringRef sr (skeletons[idx].jit_bytecode,
	                                 skeletons[idx].jit_bytecode_length);

                               if (sr.str()[0] == 'B')
                                        std::cout << "B ok\n";
                                if (sr.str()[1] == 'C')
                                        std::cout << "C ok\n";
                                if (sr.str()[2] == (char) 0xc0)
                                        std::cout << "0xc0
ok\n";
                                if (sr.str()[3] == (char) 0xde)
                                        std::cout << "0xde
ok\n";
	
	llvm::MemoryBuffer* mbjit = llvm::MemoryBuffer::getMemBufferCopy (sr);

	llvm::ErrorOr<llvm::Module*> ModuleOrErr = llvm::parseBitcodeFile (mbjit,
context);
	if (llvm::error_code EC = ModuleOrErr.getError()) {
		std::cout << ModuleOrErr.getError().message() << "\n";
	}

	Module* Mjit = ModuleOrErr.get();

	std::string eeError;
	ExecutionEngine* nee =
EngineBuilder(Mjit).setEngineKind(EngineKind::JIT).setUseMCJIT(true).setErrorStr(&eeError).create();
	if (!nee) {
		fprintf(stderr, "Could not create ExecutionEngine: %s\n",
eeError.c_str());
		assert(false);
	}

	Function f = ret_fct(Mjit);   // Function* ret_fct (Module*);    return the
function we want to jit.
	uint64_t f_ptr = nee->getFunctionAddress(f->getName());

	long (*fjited)(param_t, phi_state_t, long, long, long, long)
		= (long (*)(param_t, phi_state_t, long, long, long, long)) (intptr_t)f_ptr;

	return fjited (params, phi_state, lower, upper, inst_outer, inst_inner);
}


Thanks,
--
Willy WOLFF

On 20 Mar 2014, at 00:37, Vikas Bhargava wrote:
> segmentation fault indicates memory corruption and it's hard to tell
without seeing the exact use of the APIs. If possible, please post a complete
program and gdb stack trace from the core file. If there are multiple threads
using the global variables, please let us know.
> 
> FWIW, I have some tests to write llvm::Module to bitcode files and read
them back into llvm::Module and they work just fine with 3.4 (never tried with
tip).
> 
> thx
> vikas.
> =======> 
> 
> On Wed, Mar 19, 2014 at 2:58 PM, Willy WOLFF <willy.wolff at
etu.unistra.fr> wrote:
> all of:
> ----
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >             
llvm::MemoryBuffer::getMemBufferCopy (sr);
> ------
> string lsr = sr.str();
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >             
llvm::MemoryBuffer::getMemBuffer (lsr);
> -------
> string lsr = sr.str();
>                                 // cout << "lsr: " <<
lsr << "\n";
>                                 llvm::MemoryBuffer* mbjit >             
llvm::MemoryBuffer::getMemBufferCopy (lsr);
> 
> 
> have same result as invalid bit code.
> The result of valgrind, effectively, i have invalid reads in the
parseBitcodeFile:
> 
> ==536== Conditional jump or move depends on uninitialised value(s)
> ==536==    at 0x501FE3: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536== 
> ==536== Invalid read of size 8
> ==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==  Address 0x0 is not stack'd, malloc'd or (recently)
free'd
> ==536== 
> ==536== 
> ==536== Process terminating with default action of signal 11 (SIGSEGV)
> ==536==  Access not within mapped region at address 0x0
> ==536==    at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x501A19:
llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50AEC8: llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x50B295: llvm::parseBitcodeFile(llvm::MemoryBuffer*,
llvm::LLVMContext&) (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> ==536==    by 0x4F46C2: ??? (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> ==536==    by 0x4F2B60: main (in
/home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> 
> 
> 
> --
> Willy WOLFF
> 
> On 19 Mar 2014, at 22:11, Vikas Bhargava wrote:
> 
>> Hi Willy,
>> If the disassembly of the module works fine, then there is nothing
wrong with the module.
>> 
>> Stream uses the memorybuffer that you pass in parseBitcodeFile. If what
Will is saying is true, there is something wrong with your code in
"3:", i.e.:
>> 
>> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>>   LLVMContext& context = getGlobalContext();
>>   ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit,
context);
>>   if (error_code EC = ModuleOrErr.getError())
>>   {
>>     std::cout << ModuleOrErr.getError().
>> message() << "\n";
>>     assert(false);
>>   }
>> 
>> Can you post how you modified it in your second reply? For debugging
purpose, you can simply use MemoryBuffer::getMemBufferCopy() and not worry about
validity of stringref or null-termination. Also, you can run your program
through valgrind and check for any invalid reads.
>> 
>> HTH
>> Vikas.
>> ======>> 
>> 
>> 
>> On Wed, Mar 19, 2014 at 10:32 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr> wrote:
>> I mad the change, and still have the problem.
>> 
>> I investigate more the source code of llvm.
>> 
>> First, I change isRawBitcode function to print the content of the
parameter like this:
>> original:
http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.html#l00081
>> 
>>   inline bool isRawBitcode(const unsigned char *BufPtr,
>>                            const unsigned char *BufEnd) {
>>     // These bytes sort of have a hidden message, but it's not in
>>     // little-endian this time, and it's a little redundant.
>>           errs()<< "isRawBitcode output:\n";
>>           for (int i = 0; i < 4; i++)
>>                   errs() << BufPtr[i] << "\n";
>>           if (BufPtr != BufEnd )
>>                 errs() << "BP != BE ok\n";
>>           if (BufPtr[0] == 'B')
>>                 errs() << "B ok\n";
>>           if (BufPtr[1] == 'C')
>>                 errs() << "C ok\n";
>>           if (BufPtr[2] ==  0xc0)
>>                 errs() << "0xc0 ok\n";
>>           if (BufPtr[3] ==  0xde)
>>                 errs() << "0xde ok\n";
>> 
>>     return BufPtr != BufEnd &&
>>            BufPtr[0] == 'B' &&
>>            BufPtr[1] == 'C' &&
>>            BufPtr[2] == 0xc0 &&
>>            BufPtr[3] == 0xde;
>>   }
>> 
>> 
>> Second, I change ParseBitcodeInto as this:
>> original:
http://llvm.org/docs/doxygen/html/BitcodeReader_8cpp_source.html#l01971
>> ...
>>         errs() << "parsebitcodeinto sniff the
signature\n";
>>         uint32_t bvar = Stream.Read(8);
>>                         errs() << "B :" << bvar
<< "\n";
>>         if (bvar != 'B') {
>>                 errs() << "B :" << bvar <<
"\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>> 
>>         if (Stream.Read(8) != 'C') {
>>                 errs() << "C\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         if (  Stream.Read(8) != 0xc0 ) {
>>                 errs() << "0xc0\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         if (  Stream.Read(8) != 0xde ) {
>>                 errs() << "0xde\n";
>>                 return Error(InvalidBitcodeSignature);
>>         }
>>         // if (Stream.Read(8) != 'B' ||
>>         //     Stream.Read(8) != 'C' ||
>>         //     Stream.Read(4) != 0x0 ||
>>         //     Stream.Read(4) != 0xC ||
>>         //     Stream.Read(4) != 0xE ||
>>         //     Stream.Read(4) != 0xD
>>         //      ) {
>> ...
>> 
>> 
>> 
>> The output of the code is :
>> 
>> 
>> isRawBitcode output:
>> B
>> C
>> 
>> 
>> BP != BE ok
>> 
>> B ok
>> C ok
>> 0xc0 ok
>> 0xde ok
>> 
>> parsebitcodeinto sniff the signature
>> B :37
>> B :37
>> 
>> 
>> 
>> 
>> It's possible that Stream object is not correctly initialized?
>> 
>> 
>> On 03/13/2014 06:37 PM, Will Dietz wrote:
>> On Thu, Mar 13, 2014 at 9:02 AM, Willy WOLFF <willy.wolff at
etu.unistra.fr> wrote:
>> Hello,
>> 
>> I having a weird problem while writing a bytecode module to a string,
>> and after read/parse it for unsing on a jit.
>> 
>> I write a pass to export function to module, and put this module inside
>> a global variable.
>> I use WriteBitcodeToFile for this.
>> For debuging, after this write, I try to load the exported module with
>> parseBitcodeFile.
>> This two step works.
>> 
>> 
>> 
>> After, while the compiled program is running, I try to read and parse
>> this global variable for jiting the function.
>> 
>> 1) I read the global variable with
>>    StringRef sr (gv, gv_length);
>> 
>> 2) I manually test this bytecode by
>> (inspired by  inline bool isRawBitcode(const unsigned char *BufPtr,
>> const unsigned char *BufEnd) at
>> http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.html#l00067)
>>    if (sr.str()[0] == 'B')
>>      std::cout << "B ok\n";
>>    if (sr.str()[1] == 'C')
>>      std::cout << "C ok\n";
>>    if (sr.str()[2] == (char) 0xc0)
>>      std::cout << "0xc0 ok\n";
>>    if (sr.str()[3] == (char) 0xde)
>>      std::cout << "0xde ok\n";
>> 
>> 3) I try to parse the gv by
>>    MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>> 
>> Not sure if this is your issue, but should be fixed anyway:
>> 
>> The std::string created by "sr.str()" ends its lifetime in
this
>> statement, and MemoryBuffer for efficiency reasons
>> avoids copying data it doesn't have to (like StringRef) so will be
>> referencing the freed memory.
>> 
>> To resolve this:
>> * Pass MemoryBuffer your StringRef directly
>> * Use getMemBufferCopy()
>> * Preserve the result of sr.str() into a stack variable and pass that
>> to getMemoryBuffer() instead.
>> 
>> As a final note, check if your bitcode buffer "string" is
>> null-terminated or not.  If not, be sure to be careful and
>> do things like informing MemoryBuffer that this is the case.
>> 
>> Hope this helps,
>> ~Will
>> 
>>    LLVMContext& context = getGlobalContext();
>>    ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit,
context);
>>    if (error_code EC = ModuleOrErr.getError())
>>    {
>>      std::cout << ModuleOrErr.getError().message() <<
"\n";
>>      assert(false);
>>    }
>> 
>> 
>> 
>> 
>> This is the execution result:
>> B ok
>> C ok
>> 0xc0 ok
>> 0xde ok
>> Invalid bitcode signature
>> 
>> 
>> 
>> Ok is not working :/
>> But why ???
>> 
>> 
>> 
>> For debuging, between 2) and 3), I export the readed module and write
to
>> a file on my hard drive,
>> and try llvm-dis, and the dissasembly of the module works.
>> 
>> Wath's wrong? Any idea for solve this problem?
>> 
>> Thanks you very much.
>> 
>> Regards,
>> Willy
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140320/02e6b9c9/attachment.html>

Reasonably Related Threads

Search for more possibly parallel threads

llvm dev - Mar 2014 - [LLVMdev] load bytecode from string for jiting problem

[LLVMdev] load bytecode from string for jiting problem

[LLVMdev] load bytecode from string for jiting problem

[LLVMdev] load bytecode from string for jiting problem

Reasonably Related Threads