Hi Eli,
Thanks for the reply. I tried with -Xlinker="-ldl ". However it does
not
seem to make a difference. It seems that when bugpoint is run with
--run-jit, the linker args are not passed to gcc (from
tools/bugpoint/ExecutionDriver.cpp) :
if (InterpreterSel == RunLLC || InterpreterSel == RunCBE ||
InterpreterSel == CBE_bug || InterpreterSel == LLC_Safe)
RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile,
OutputFile, AdditionalLinkerArgs,
SharedObjs,
Timeout, MemoryLimit);
else
RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile,
OutputFile, std::vector<std::string>(),
SharedObjs, Timeout, MemoryLimit);
I tried the following after this:
(1) Firstly instead of running Gap (
http://www.gap-system.org/Download/UNIXInst.html), I am now trying to run
python with lli (http://www.python.org/download/releases/2.5.2/). I managed
to compile python.bc and here again I face the same problem:
llc and gcc can get python.exe to run (which is great :)!) :
$ llc -f python.bc
$ gcc -o python.exe python.s -ldl -lutil -lm -lrt
$ ./python.exe
Python 2.5.2 (r252:60911, Oct 31 2008, 14:41:11)
[GCC 4.2.1 (Based on Apple Inc. build 5623) (LLVM build)] on linux2
Type "help", "copyright", "credits" or
"license" for more information.>>>
however, when I try to run python.bc using lli it crashes with a
segmentation fault:
$ lli -load=/usr/lib/libdl.so -load=/usr/lib/libutil.so
-load=/usr/lib/libm.so -load=/usr/lib/librt.so python.bc
When i try it with gdb, it seems that the crash is somewhere inside python
code (since bt only shows ?? ). Before the crash, I could see that the
memory consumption(VM) reaches somewhere near 80% of my 2GB RAM (seen via
top, and that too a sudden increase from around when it was previously
occupying around 2-3% of VM). I tried to run this on a 64-bit machine which
has 8GB RAM and still have the same issue wrt memory.
(2) Finally I wrote a pass (and loaded it through opt) to instrument each
function's (in python code ) entry and exit and then ran the instrumented
program with both [llc ; gcc] combination and lli. In the lli version a
single method (subtype_traverse) is recursively called (about 2 million
times) until the program runs out of memory while the statically compiled
code (llc + gcc) calls this method (I am comparing the calls in the same
context in both cases) only once:
python with llc + gcc :
...
(tupletraverse (visit_decref (type_is_gc)))
(subtype_traverse (visit_decref (type_is_gc))
(type_traverse (visit_decref) (visit_decref) ...
python with lli:
(tupletraverse (visit_decref (type_is_gc)))
(subtype_traverse (visit_decref (type_is_gc)) (subtype_traverse
(visit_decref (type_is_gc))(subtype_traverse (visit_decref (type_is_gc))
..... about 2 million times
Looking at the code (Objects/typeobject.c:
http://google.com/codesearch?hl=en&q=show:VK_wUSuAZto:jHKC99mjNVM:4z02hQcYQRY&sa=N&ct=rd&cs_p=http://gentoo.osuosl.org/distfiles/Python-2.5.tar.bz2&cs_f=Python-2.5/Objects/typeobject.c
)
it seems the last call (through a function pointer) in subtype_traverse
results in this never-ending recursive call.
Has anyone tried compiling python to bit code and running it the LLVM JIT
before ?
Thanks for your time.
- Prakash
On Tue, Oct 28, 2008 at 3:02 PM, Eli Friedman <eli.friedman at
gmail.com>wrote:
> On Tue, Oct 28, 2008 at 12:17 PM, Prakash Prabhu
> <prakash.prabhu at gmail.com> wrote:
> > Generating reference output from raw program: <cbe><gcc>
> > Error running tool:
> [snip]
> > /tmp/cc08IpX8.o: In function `SyLoadModule':
> > bugpoint-test-program.bc.cbe.c:(.text+0x25705): undefined reference to
> > `dlopen'
> [snip]
>
> This is saying that compilation with CBE is failing. Try something
> like -Xlinker -ldl?
>
> -Eli
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081102/54161ee7/attachment.html>
Hi Prakash, Unfortunately it looks like you need to do quite a bit of investigation into this. However, I hope I can provide some useful tips. 1. In general, lli and llc generate exact the same code except lli default to static codegen while llc defaults to dynamic-no-pic codegen. So try passing -relocation-model=dynamic-no-pic to lli. If this works, that means there are issues with static codegen. 2. It could be a JIT encoding bug. If you can identify a problematic function, it's possible examine the generated code in gdb and compare it with llc generated assembly. 3. It could be a bug in the app and it's exposed when running under the JIT. You can try enabling additional debugging output. Hope this helps. Evan On Nov 2, 2008, at 2:55 PM, Prakash Prabhu wrote:> Hi Eli, > > Thanks for the reply. I tried with -Xlinker="-ldl ". However it does > not seem to make a difference. It seems that when bugpoint is run > with --run-jit, the linker args are not passed to gcc (from tools/ > bugpoint/ExecutionDriver.cpp) : > > if (InterpreterSel == RunLLC || InterpreterSel == RunCBE || > InterpreterSel == CBE_bug || InterpreterSel == LLC_Safe) > > RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile, > OutputFile, AdditionalLinkerArgs, > SharedObjs, > Timeout, MemoryLimit); > > else > > > RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile, > OutputFile, > std::vector<std::string>(), > SharedObjs, Timeout, MemoryLimit); > > > I tried the following after this: > > (1) Firstly instead of running Gap (http://www.gap-system.org/Download/UNIXInst.html > ), I am now trying to run python with lli (http://www.python.org/download/releases/2.5.2/ > ). I managed to compile python.bc and here again I face the same > problem: > > llc and gcc can get python.exe to run (which is great :)!) : > > $ llc -f python.bc > $ gcc -o python.exe python.s -ldl -lutil -lm -lrt > $ ./python.exe > Python 2.5.2 (r252:60911, Oct 31 2008, 14:41:11) > [GCC 4.2.1 (Based on Apple Inc. build 5623) (LLVM build)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> > > however, when I try to run python.bc using lli it crashes with a > segmentation fault: > > $ lli -load=/usr/lib/libdl.so -load=/usr/lib/libutil.so -load=/usr/ > lib/libm.so -load=/usr/lib/librt.so python.bc > > When i try it with gdb, it seems that the crash is somewhere inside > python code (since bt only shows ?? ). Before the crash, I could > see that the memory consumption(VM) reaches somewhere near 80% of my > 2GB RAM (seen via top, and that too a sudden increase from around > when it was previously occupying around 2-3% of VM). I tried to run > this on a 64-bit machine which has 8GB RAM and still have the same > issue wrt memory. > > (2) Finally I wrote a pass (and loaded it through opt) to instrument > each function's (in python code ) entry and exit and then ran the > instrumented program with both [llc ; gcc] combination and lli. In > the lli version a single method (subtype_traverse) is recursively > called (about 2 million times) until the program runs out of memory > while the statically compiled code (llc + gcc) calls this method (I > am comparing the calls in the same context in both cases) only once: > > python with llc + gcc : > ... > (tupletraverse (visit_decref (type_is_gc))) > (subtype_traverse (visit_decref (type_is_gc)) > (type_traverse (visit_decref) (visit_decref) ... > > python with lli: > > (tupletraverse (visit_decref (type_is_gc))) > (subtype_traverse (visit_decref (type_is_gc)) (subtype_traverse > (visit_decref (type_is_gc))(subtype_traverse (visit_decref > (type_is_gc)) ..... about 2 million times > > Looking at the code (Objects/typeobject.c: http://google.com/codesearch?hl=en&q=show:VK_wUSuAZto:jHKC99mjNVM:4z02hQcYQRY&sa=N&ct=rd&cs_p=http://gentoo.osuosl.org/distfiles/Python-2.5.tar.bz2&cs_f=Python-2.5/Objects/typeobject.c) > > it seems the last call (through a function pointer) in > subtype_traverse results in this never-ending recursive call. > > Has anyone tried compiling python to bit code and running it the > LLVM JIT before ? > > Thanks for your time. > > - Prakash > > > > > > On Tue, Oct 28, 2008 at 3:02 PM, Eli Friedman > <eli.friedman at gmail.com> wrote: > On Tue, Oct 28, 2008 at 12:17 PM, Prakash Prabhu > <prakash.prabhu at gmail.com> wrote: > > Generating reference output from raw program: <cbe><gcc> > > Error running tool: > [snip] > > /tmp/cc08IpX8.o: In function `SyLoadModule': > > bugpoint-test-program.bc.cbe.c:(.text+0x25705): undefined > reference to > > `dlopen' > [snip] > > This is saying that compilation with CBE is failing. Try something > like -Xlinker -ldl? > > -Eli > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081102/85e1ce43/attachment.html>
Hi Evan,
Thanks for the pointers. We found a simple test case that causes the problem
(thanks to Tom in my group):
#include<stdio.h>
#include<stdlib.h>
void test();
void (*funcPtr)();
int main(int argc, char **argv) {
funcPtr = test;
test();
}
void test() {
if(funcPtr == test) {
printf("OK!\n");
} else {
fprintf(stderr, "Bad!\n");
exit(1);
}
}
$ llvm-gcc -emit-llvm -o FPtrEqTest.bc -c FPtrEqTest.c
$ llc -f FPtrEqTest.bc
$ gcc -o FPtrEqTest FPtrEqTest.s
$ ./FPtrEqTest
OK!
$ lli FPtrEqTest.bc
Bad!
The above test case is just a smaller version of the one in Python's
subtype_traverse which also tests a function pointer and calls itself. It
seems the problem arises due comparison with the stub's address when a
comparison with the actual address of the compiled function is intended.
thanks,
Prakash
On Mon, Nov 3, 2008 at 2:07 AM, Evan Cheng <evan.cheng at apple.com>
wrote:
> Hi Prakash,
> Unfortunately it looks like you need to do quite a bit of investigation
> into this. However, I hope I can provide some useful tips.
>
> 1. In general, lli and llc generate exact the same code except lli default
> to static codegen while llc defaults to dynamic-no-pic codegen. So try
> passing -relocation-model=dynamic-no-pic to lli. If this works, that means
> there are issues with static codegen.
> 2. It could be a JIT encoding bug. If you can identify a problematic
> function, it's possible examine the generated code in gdb and compare
it
> with llc generated assembly.
> 3. It could be a bug in the app and it's exposed when running under the
> JIT. You can try enabling additional debugging output.
>
> Hope this helps.
>
> Evan
>
> On Nov 2, 2008, at 2:55 PM, Prakash Prabhu wrote:
>
> Hi Eli,
>
> Thanks for the reply. I tried with -Xlinker="-ldl ". However it
does not
> seem to make a difference. It seems that when bugpoint is run with
> --run-jit, the linker args are not passed to gcc (from
> tools/bugpoint/ExecutionDriver.cpp) :
>
> if (InterpreterSel == RunLLC || InterpreterSel == RunCBE ||
> InterpreterSel == CBE_bug || InterpreterSel == LLC_Safe)
>
> RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile,
> OutputFile, AdditionalLinkerArgs,
> SharedObjs,
> Timeout, MemoryLimit);
>
> else
>
>
> RetVal = AI->ExecuteProgram(BitcodeFile, InputArgv, InputFile,
> OutputFile,
std::vector<std::string>(),
> SharedObjs, Timeout, MemoryLimit);
>
>
> I tried the following after this:
>
> (1) Firstly instead of running Gap (
> http://www.gap-system.org/Download/UNIXInst.html), I am now trying to run
> python with lli (http://www.python.org/download/releases/2.5.2/). I
> managed to compile python.bc and here again I face the same problem:
>
> llc and gcc can get python.exe to run (which is great :)!) :
>
> $ llc -f python.bc
> $ gcc -o python.exe python.s -ldl -lutil -lm -lrt
> $ ./python.exe
> Python 2.5.2 (r252:60911, Oct 31 2008, 14:41:11)
> [GCC 4.2.1 (Based on Apple Inc. build 5623) (LLVM build)] on linux2
> Type "help", "copyright", "credits" or
"license" for more information.
> >>>
>
> however, when I try to run python.bc using lli it crashes with a
> segmentation fault:
>
> $ lli -load=/usr/lib/libdl.so -load=/usr/lib/libutil.so
> -load=/usr/lib/libm.so -load=/usr/lib/librt.so python.bc
>
> When i try it with gdb, it seems that the crash is somewhere inside python
> code (since bt only shows ?? ). Before the crash, I could see that the
> memory consumption(VM) reaches somewhere near 80% of my 2GB RAM (seen via
> top, and that too a sudden increase from around when it was previously
> occupying around 2-3% of VM). I tried to run this on a 64-bit machine which
> has 8GB RAM and still have the same issue wrt memory.
>
> (2) Finally I wrote a pass (and loaded it through opt) to instrument each
> function's (in python code ) entry and exit and then ran the
instrumented
> program with both [llc ; gcc] combination and lli. In the lli version a
> single method (subtype_traverse) is recursively called (about 2 million
> times) until the program runs out of memory while the statically compiled
> code (llc + gcc) calls this method (I am comparing the calls in the same
> context in both cases) only once:
>
> python with llc + gcc :
> ...
> (tupletraverse (visit_decref (type_is_gc)))
> (subtype_traverse (visit_decref (type_is_gc))
> (type_traverse (visit_decref) (visit_decref) ...
>
> python with lli:
>
> (tupletraverse (visit_decref (type_is_gc)))
> (subtype_traverse (visit_decref (type_is_gc)) (subtype_traverse
> (visit_decref (type_is_gc))(subtype_traverse (visit_decref (type_is_gc))
> ..... about 2 million times
>
> Looking at the code (Objects/typeobject.c:
>
http://google.com/codesearch?hl=en&q=show:VK_wUSuAZto:jHKC99mjNVM:4z02hQcYQRY&sa=N&ct=rd&cs_p=http://gentoo.osuosl.org/distfiles/Python-2.5.tar.bz2&cs_f=Python-2.5/Objects/typeobject.c
> )
>
> it seems the last call (through a function pointer) in subtype_traverse
> results in this never-ending recursive call.
>
> Has anyone tried compiling python to bit code and running it the LLVM JIT
> before ?
>
> Thanks for your time.
>
> - Prakash
>
>
>
>
>
> On Tue, Oct 28, 2008 at 3:02 PM, Eli Friedman <eli.friedman at
gmail.com>wrote:
>
>> On Tue, Oct 28, 2008 at 12:17 PM, Prakash Prabhu
>> <prakash.prabhu at gmail.com> wrote:
>> > Generating reference output from raw program:
<cbe><gcc>
>> > Error running tool:
>> [snip]
>> > /tmp/cc08IpX8.o: In function `SyLoadModule':
>> > bugpoint-test-program.bc.cbe.c:(.text+0x25705): undefined
reference to
>> > `dlopen'
>> [snip]
>>
>> This is saying that compilation with CBE is failing. Try something
>> like -Xlinker -ldl?
>>
>> -Eli
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081103/5aab341b/attachment.html>