Gaël Thomas
2014-Sep-09 23:42 UTC
[LLVMdev] VMKit is retired (but you can help if you want!)
Oups, sorry for the mistake, llcj (not llc:)) is not more maintained! Gaël Le 10 sept. 2014 00:27, "Gaël Thomas" <gael.thomas00 at gmail.com> a écrit :> Hi Brian, > > So, I confirm, llc is not more maintained. And using vmjc is probably > the good starting point to translate Java bytecode into llvm bitcode. > > However, I think that your hack (changing the way VMKitGCPrinter.cpp > resolves symbols) is probably not the good way to solve the initial > problem. And maybe that your current problem is simply a consequence > of the mangling problem. So, let's start with the first problem :) > > So, I answer in your previous mail. As it's the beginning of the year > (I'm teaching), I haven't found any time to install vmkit and make the > tests. But we will proceed step by step to understand the problem. In > parallel, I'm also writing a documentation to describe the internals > of VMKit, it will be useful for you and other people interested by the > AOT. > > 2014-09-02 18:13 GMT+02:00 Brian Faull <bfaull at cog-e.com>: > > Hi Gaël, > > > > So as not to get too far into the "weeds" -- let's start with minimal > configuration, then `vmjc` (.class -> .bc), then `llc` (.bc -> asm) to make > sure I'm on the right path! > > > > 1- CONFIGURATION > > I can configure out of the box with no problem. But perhaps I need to > configure differently. > > > > I start with a clean checkout of the VMKit repository into > /path/to/vmkit_test-clean . I'm on x86_64 using OpenJDK1.6 as follows: > > > > =========> > /path/to/vmkit_test-clean > > ./configure --with-llvm-config-path=/path/to/llvm33/bin/llvm-config > --with-classpath-impl=openjdk > --with-openjdk-path=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64 > > =========> > > > I have PATH set to provide `javac` as > /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/javac and > LD_LIBRARY_PATH to include > /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64 and > ...amd64/server . > > > > This `configure` command works. I see two WARNINGs during this build, > as a result of > > =========> > [vmkit ./mmtk/java]: Linking module 'FinalMMTk.bc' > > /path/to/llvm33/bin/llvm-link > /path/to/vmkit_test-clean/mmtk/java/Release+Asserts/mmtk-vmkit.bc > /path/to/vmkit_test-clean/Release+Asserts/lib/MMTKAlloc.bc > /path/to/vmkit_test-clean/Release+Asserts/lib/MMTKRuntime.bc -o > /path/to/vmkit_test-clean/mmtk/java/Release+Asserts/FinalMMTk.bc > > WARNING: Linking two modules of different data layouts! > > WARNING: Linking two modules of different data layouts! > > =========> > I'm not sure if that matters. > > It does not matter :) > > > > > > I have also tried OpenJDK 1.7 and GNU Classpath, and I think I see the > same results, but perhaps I didn't reset environment variables or something. > > > > Some fundamental questions: basically, how do *YOU* configure VMKit? > > -> Which version of Java do you recommend? 1.5, 1.6, 1.7...? Is there > any benefit of using one or another, or any restrictions? > > -> Do you recommend OpenJDK or GNU Classpath? Is there any benefit of > using one or the other, or any tradeoff? > > You only have to use openjdk (gnu classpath is not more maintained as > the project is also retired:)). We have a bug with the new versions of > openjdk, and we are using the 6u23. I don't know if all the previous > versions will be ok. So, I suggest using this version. > > > > > > 2- COMPILATION: `vmjc` and `llc` to for Java class to native assembly. > > I do the following, with the same OpenJDK 1.6.0 `javac` command: > > > > =========> > cd /path/to/vmkit_test-clean/tools/trainer > > javac HelloWorld.java > > ../../Release+Asserts/bin/vmjc -print-aot-stats HelloWorld.class > > /path/to/llvm33/bin/llc > -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.bc > > =========> > > > The `javac` command succeeds, and produces Java classfile > HelloWorld.class. `java HelloWorld` outputs "Hello World". > > The `vmjc` command succeeds, and produces LLVM bitcode > HelloWorld.class.bc. `/path/to/llvm33/bin/llvm-nm HelloWorld.class.bc` > shows reasonable symbols from that bitcode. > > The `llc` command fails exactly the same as Dave Brazdil reported on 08 > Mar; http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-March/070995.html . > > => It seems that I need to point to that shared-object using > -load=../../Release+Asserts/lib/static-gc-printer.so . I found this in a > previous post to the LLVM list. Is this correct? > > Yes, it's mandatory. > > > => The output I see is as follows: > > =========> > llc: VmkitGCPrinter.cpp:268: llvm::Constant *FindMetadata(const > llvm::Function &): Assertion `0 && "Should have found a JavaMethod"' failed. > > 0 llc 0x000000000154d5ee > llvm::sys::PrintStackTrace(_IO_FILE*) + 46 > > 1 llc 0x000000000154d8ab > > 2 llc 0x000000000154db1e > > 3 libpthread.so.0 0x0000003f31a0f710 > > 4 libc.so.6 0x0000003ee9232925 gsignal + 53 > > 5 libc.so.6 0x0000003ee9234105 abort + 373 > > 6 libc.so.6 0x0000003ee922ba4e > > 7 libc.so.6 0x0000003ee922bb10 __assert_perror_fail + 0 > > 8 static-gc-printer.so 0x00007febfc02b135 FindMetadata(llvm::Function > const&) + 3813 > > 9 static-gc-printer.so 0x00007febfc02b892 > > 10 llc 0x0000000000e620a5 > llvm::AsmPrinter::doFinalization(llvm::Module&) + 2597 > > 11 llc 0x00000000014bf0dc > llvm::FPPassManager::doFinalization(llvm::Module&) + 92 > > 12 llc 0x00000000014bf58e > llvm::MPPassManager::runOnModule(llvm::Module&) + 1134 > > 13 llc 0x00000000014bfb4e > llvm::PassManagerImpl::run(llvm::Module&) + 302 > > 14 llc 0x00000000014bfda1 > llvm::PassManager::run(llvm::Module&) + 33 > > 15 llc 0x00000000005ca548 > > 16 llc 0x00000000005c9242 main + 226 > > 17 libc.so.6 0x0000003ee921ed1d __libc_start_main + 253 > > 18 llc 0x00000000005c8f49 > > Stack dump: > > 0. Program arguments: /path/to/llvm33/bin/llc > -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.bc > > Aborted (core dumped) > > =========> > => Do you see something similar? Or what could be the configuration > difference? > > > > I fixed this situation by modifying the sources backing > static-gc-printer.so, as shown below. > > > > Are we on the right path? Is this a good start? > > It's a very good start :) But your hack is not the good solution > because you should return from FindMetaData at the line 229 (you > should not reach the test at line 239). > > So, can you try this compilation scheme (without your hack in > StaticGCPrinter.cpp:))? > > vmjc -print-aot-stats HelloWorld.class > opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass > HelloWorld.class.bc -o HelloWorld.class.gc.bc > llc -load=../../Release+Asserts/lib/static-gc-printer.so > HelloWorld.class.gc.bc > > If it does not change anything, can you send me the > HelloWorld.class.bc file? I will quickly see if everything is ok for > the GC and where is the problem. > > See you, > Gaël > > > > > Thank you again for your help. I really appreciate it. > > =brian > > > > > > > > > 2014-09-09 21:51 GMT+02:00 Brian Faull <bfaull at cog-e.com>: > > Hello again Gaël, (et al) > > > > More on rekindling work on VMKit! Thank you for your interactions thus > far on- and off-list. > > > > As you suggested in your VMKit-retirement email (to which I'm attempting > to respond), I'm interested in producing a Java-to-LLVM compiler out of > VMKit. I'd like to take you up on your offer to help understand the > architecture. If I can get the a Java-to-LLVM compiler working for my > purposes, I'll be happy to maintain at least this part of the VMKit project. > > > > To that end, I have exactly one fundamental question to start: > > > > > > How do you suggest to perform Java-to-LLVM compilation using VMKit? > > > > > > Here are my thoughts on this: > > > > It looks like the `llcj` tool (VMKit/tools/llcj/llcj.cpp) was meant for > this; it declares itself as "Java to native compiler" but it appears that > it hasn't been maintained (e.g., deprecated interfaces, no-longer-existent > command-line options and libraries). `llcj` is described in Appendix A of > Geoffray's thesis, > http://pagesperso-systeme.lip6.fr/Nicolas.Geoffray/these-geoffray.pdf > linked from the VMKit site. > > > > `llcj` attempts the following to translate Java to native: > > * vmjc (Java .class => LLVM bitcode .bc) > > * opt (LLVM bitcode optimizer) > > * llc (LLVM bitcode => target assembly .s) > > * gcc (assemble and link) > > > > Is that "The Right Way"? > > > > After some time hacking various changes, I can use basically that method > to compile VMKit/tools/trainer/HelloWorld.java into a linked executable > (but it barfs at runtime and I can't fix it). > > > > I'm doing roughly the following, on x86_64, using LLVM3.3, OpenJDK 1.6.0 > build 30, a debug build of VMKit (required small build-hacks, which I could > describe on request), and the (perhaps-incorrect) edit to > `static-gc-printer.so` described in an earlier post: > http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/076128.html : > > cd /path/to/vmkit/tools/trainer > > javac HelloWorld.java > > ../../Debug+Asserts/bin/vmjc -main=HelloWorld -print-aot-stats > HelloWorld.class > > /path/to/llvm33/bin/llc > -load=../../Debug+Asserts/lib/static-gc-printer.so HelloWorld.class.bc > > > > That produces for me a reasonable native assembly file. > > > > Next I assemble and link, using a set of libraries/objects inspired by > `llcj` and refined by a semi-automated trial-and-error: > > /path/to/llvm33/bin/clang++ -g3 -O0 -o HelloWorld.class.exe > HelloWorld.class.sed.s -L/path/to/vmkit_test-codechanges/Debug+Asserts/lib > -L/path/to/llvm33/lib -pthread -lm -ldl -lz -lJ3 -lClasspath > -lJ3Compiler -lCommonThread -lVmkit -lVmkitCompiler -lPrecompiled > -lFinalMMTk -lLLVMX86Info -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Utils > -lLLVMX86AsmPrinter -lLLVMExecutionEngine -lLLVMScalarOpts -lLLVMipo > -lLLVMipa -lLLVMAnalysis -lLLVMInstCombine -lLLVMInstrumentation > -lLLVMTarget -lLLVMJIT -lLLVMRuntimeDyld -lLLVMObject -lLLVMObjCARCOpts > -lLLVMVectorize -lLLVMTransformUtils -lLLVMSelectionDAG -lLLVMCodeGen > -lLLVMMC -lLLVMCore -lLLVMSupport -rdynamic > > > > This links and executes, but here are my problems/hacks: > > * `llc` generates an assembly file with symbols (e.g., ____Vstatic_buf() > and friends) that conflict with one of the archives listed above > (libPrecompiled.a, which I believe is necessary for java_lang_Object and > others). I basically remove those symbols (sed '/_buf$/ s/globl/weak/' > HelloWorld.class.s > HelloWorld.class.sed.s) before assembling. I don't > know if this is a reasonable hack or if it's causing problems. > > * I get the following error at runtime (because some fields are NULL) > and I can't make progress: > > HelloWorld.class.exe: JavaRuntimeJIT.cpp:381: void > *j3ResolveVirtualStub(j3::JavaObject *): Assertion `FI->Metadata != __null > && "Wrong stack trace"' failed. > > > > Am I on the right path, or should I be considering other methods? Or > wrong entirely? > > > > Ultimately, I propose changing `llcj` into something like a Python > script that executes the appropriate utilities to start from Java source > and resulting in a linked target-architecture executable, using `clang++` > as compiler-driver for assembling and linking. It would be essentially a > work-alike to `gcj`. > > > > Can you help point me in the right direction? > > > > Thank you, > > Brian > > > > > > > > > > > > Date: Mon, 1 Sep 2014 21:34:58 +0200 > > From: Gaël Thomas <gael.thomas00 at gmail.com> > > To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>, Nicolas > > Geoffray <nicolas.geoffray at gmail.com> > > Subject: [LLVMdev] VMKit is retired (but you can help if you want!) > > Message-ID: > > <CAOWuPDcZBpt_JJ5yo5YN=C+RWbtbneXB1UGd90d0mXdnrs8> RQ at mail.gmail.com> > > Content-Type: text/plain; charset=UTF-8 > > > > Hi all, > > > > So, as explained in the LLVM weekly, the VMKit project is retired. It > > was a very fun project, but we don't have any manpower to maintain the > > code since one year. If someone is interested by restarting the > > project, just send me an email, I can help to understand the > > architecture of the project and how it works. I'm pretty sure that we > > can also extract a Java to LLVM compiler easily (I know that some > > people are interested by that). > > > > And I want to thank all the LLVM team for their support during these > > last ten (yes ten:)) years! Without their help, it would have been > > impossible to develop VMKit. > > > > See you and thank you! > > Gaël > > > > > > > > > > > > -- > ------------------------------------------------------------------- > Gaël Thomas, Associate Professor, UPMC > http://pagesperso-systeme.lip6.fr/Gael.Thomas/ > ------------------------------------------------------------------- >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140910/1250d6c8/attachment.html>
Brian Faull
2014-Sep-10 18:24 UTC
[LLVMdev] VMKit is retired (but you can help if you want!)
Okay -- thank you for the suggestions, but without my hack (which I will happily
abandon! :) I'm still getting the error "Should have found a
JavaMethod". I'll describe what I'm doing and attach the bitcode
as you requested.
Here's what I'm doing, if you're able to recreate:
* installed OpenJDK 6u23
* clean version of VMKit from the repository, configured as follows:
./configure --with-llvm-config-path=/path/to/llvm33/bin/llvm-config
--with-classpath-impl=openjdk --with-openjdk-path=/path/to/jdk1.6.0_23
where directory /path/to/jdk1.6.0_23 contains the installation of OpenJDK6u23,
e.g., bin, jre, lib, include.
* your class-to-bitcode commands, which includes several things I hadn't
tried. Specifically, I'm invoking as follows:
vmjc -print-aot-stats HelloWorld.class
opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass -o
HelloWorld.class.gc.bc HelloWorld.class.bc
llc -load=../../Release+Asserts/lib/static-gc-printer.so
HelloWorld.class.gc.bc
The last command fails with the "Should have found a JavaMethod"
error. The only difference from yours is the order of arguments to `opt`, just
to strictly follow the `opt [options] <input bitcode file>` usage.
I think the `vmjc` command needs additionally the "-main=" argument,
but I have tried both ways with the same outcome.
Further comments inline below, much snipping.
On Sep 9, 2014, at 7:42 PM, Gaël Thomas <gael.thomas00 at gmail.com>
wrote:> Oups, sorry for the mistake, llcj (not llc:)) is not more maintained!
>
> Gaël
>
> Le 10 sept. 2014 00:27, "Gaël Thomas" <gael.thomas00 at
gmail.com> a écrit :
> Hi Brian,
>
...>
> So, I answer in your previous mail. As it's the beginning of the year
> (I'm teaching), I haven't found any time to install vmkit and make
the
> tests. But we will proceed step by step to understand the problem. In
> parallel, I'm also writing a documentation to describe the internals
> of VMKit, it will be useful for you and other people interested by the
> AOT.
Fantastic; such a document would be very helpful; thank you. And good luck with
the teaching!
> You only have to use openjdk (gnu classpath is not more maintained as
> the project is also retired:)). We have a bug with the new versions of
> openjdk, and we are using the 6u23. I don't know if all the previous
> versions will be ok. So, I suggest using this version.
I found OpenJDK6u23 somewhat difficult to find; for others searching it's at
http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase6-419409.html#jdk-6u23-oth-JPR
I had previously not found a version this old at
https://java.net/projects/openjdk6/downloads
>
> > => The output I see is as follows:
> > =========> > llc: VmkitGCPrinter.cpp:268: llvm::Constant
*FindMetadata(const llvm::Function &): Assertion `0 && "Should
have found a JavaMethod"' failed.
...> > Are we on the right path? Is this a good start?
>
> It's a very good start :) But your hack is not the good solution
> because you should return from FindMetaData at the line 229 (you
> should not reach the test at line 239).
Good start! This seems to be the problem, because I seem to be in the wrong
code path.
>
> So, can you try this compilation scheme (without your hack in
> StaticGCPrinter.cpp:))?
>
> vmjc -print-aot-stats HelloWorld.class
> opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass
> HelloWorld.class.bc -o HelloWorld.class.gc.bc
> llc -load=../../Release+Asserts/lib/static-gc-printer.so
HelloWorld.class.gc.bc
With the commands as you suggest, I step through the execution of the `llc`
command in the debugger. When the HelloWorld::main() function is passed to
FindMetadata(), the F.getName() is
"JnJVM_HelloWorld_main___3Ljava_lang_String_2" and F.UseList is NULL
(wrong, I think). This forces the code path to drop immediately out of the
first for() loop (past the return at VmkitGCPrinter.cpp:229, and on to the block
at :239). In contrast, when the <init> function (F.getName() is
"JnJVM_HelloWorld__0003Cinit_0003E__") is processed in FindMetadata(),
F.UseList is non-NULL.
>
> If it does not change anything, can you send me the
> HelloWorld.class.bc file? I will quickly see if everything is ok for
> the GC and where is the problem.
The bitcode is (effectively) identical before and after the `opt` line, which is
presumably not correct. `llvm-diff` reports no differences, and a `diff` of the
`llvm-dis` output reports only that the module names are different. I will
attach the file off-list because I expect it will get scrubbed anyway.
The above is with your exact commands. Adding the "-main=HelloWorld"
argument to `vmjc` puts 3 additional symbols in the bitcode:
U StartJnjvmWithoutJIT
T main
d mainClass
That is, adds a main() in addition to the HelloWorld::main() (which gets mangled
as "JnJVM_HelloWorld_main___3Ljava_lang_String_2"). I think this is
necessary and correct for non-JIT linking and execution, but that's perhaps
further down the line.
I am continuing to look at the situation, but I would appreciate hearing
anything that you can see from the bitcode file.
Thank you again!
Brian
>
>
> 2014-09-09 21:51 GMT+02:00 Brian Faull <bfaull at cog-e.com>:
> > Hello again Gaël, (et al)
> >
> > More on rekindling work on VMKit! Thank you for your interactions thus
far on- and off-list.
> >
> > As you suggested in your VMKit-retirement email (to which I'm
attempting to respond), I'm interested in producing a Java-to-LLVM compiler
out of VMKit. I'd like to take you up on your offer to help understand the
architecture. If I can get the a Java-to-LLVM compiler working for my purposes,
I'll be happy to maintain at least this part of the VMKit project.
> >
> > To that end, I have exactly one fundamental question to start:
> >
> >
> > How do you suggest to perform Java-to-LLVM compilation using VMKit?
> >
> >
> > Here are my thoughts on this:
> >
> > It looks like the `llcj` tool (VMKit/tools/llcj/llcj.cpp) was meant
for this; it declares itself as "Java to native compiler" but it
appears that it hasn't been maintained (e.g., deprecated interfaces,
no-longer-existent command-line options and libraries). `llcj` is described in
Appendix A of Geoffray's thesis,
http://pagesperso-systeme.lip6.fr/Nicolas.Geoffray/these-geoffray.pdf linked
from the VMKit site.
> >
> > `llcj` attempts the following to translate Java to native:
> > * vmjc (Java .class => LLVM bitcode .bc)
> > * opt (LLVM bitcode optimizer)
> > * llc (LLVM bitcode => target assembly .s)
> > * gcc (assemble and link)
> >
> > Is that "The Right Way"?
> >
> > After some time hacking various changes, I can use basically that
method to compile VMKit/tools/trainer/HelloWorld.java into a linked executable
(but it barfs at runtime and I can't fix it).
> >
> > I'm doing roughly the following, on x86_64, using LLVM3.3, OpenJDK
1.6.0 build 30, a debug build of VMKit (required small build-hacks, which I
could describe on request), and the (perhaps-incorrect) edit to
`static-gc-printer.so` described in an earlier post:
http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/076128.html :
> > cd /path/to/vmkit/tools/trainer
> > javac HelloWorld.java
> > ../../Debug+Asserts/bin/vmjc -main=HelloWorld -print-aot-stats
HelloWorld.class
> > /path/to/llvm33/bin/llc
-load=../../Debug+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
> >
> > That produces for me a reasonable native assembly file.
> >
> > Next I assemble and link, using a set of libraries/objects inspired by
`llcj` and refined by a semi-automated trial-and-error:
> > /path/to/llvm33/bin/clang++ -g3 -O0 -o HelloWorld.class.exe
HelloWorld.class.sed.s -L/path/to/vmkit_test-codechanges/Debug+Asserts/lib
-L/path/to/llvm33/lib -pthread -lm -ldl -lz -lJ3 -lClasspath -lJ3Compiler
-lCommonThread -lVmkit -lVmkitCompiler -lPrecompiled -lFinalMMTk
-lLLVMX86Info -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Utils -lLLVMX86AsmPrinter
-lLLVMExecutionEngine -lLLVMScalarOpts -lLLVMipo -lLLVMipa -lLLVMAnalysis
-lLLVMInstCombine -lLLVMInstrumentation -lLLVMTarget -lLLVMJIT -lLLVMRuntimeDyld
-lLLVMObject -lLLVMObjCARCOpts -lLLVMVectorize -lLLVMTransformUtils
-lLLVMSelectionDAG -lLLVMCodeGen -lLLVMMC -lLLVMCore -lLLVMSupport -rdynamic
> >
> > This links and executes, but here are my problems/hacks:
> > * `llc` generates an assembly file with symbols (e.g.,
____Vstatic_buf() and friends) that conflict with one of the archives listed
above (libPrecompiled.a, which I believe is necessary for java_lang_Object and
others). I basically remove those symbols (sed '/_buf$/ s/globl/weak/'
HelloWorld.class.s > HelloWorld.class.sed.s) before assembling. I don't
know if this is a reasonable hack or if it's causing problems.
> > * I get the following error at runtime (because some fields are NULL)
and I can't make progress:
> > HelloWorld.class.exe: JavaRuntimeJIT.cpp:381: void
*j3ResolveVirtualStub(j3::JavaObject *): Assertion `FI->Metadata != __null
&& "Wrong stack trace"' failed.
> >
> > Am I on the right path, or should I be considering other methods? Or
wrong entirely?
> >
> > Ultimately, I propose changing `llcj` into something like a Python
script that executes the appropriate utilities to start from Java source and
resulting in a linked target-architecture executable, using `clang++` as
compiler-driver for assembling and linking. It would be essentially a work-alike
to `gcj`.
> >
> > Can you help point me in the right direction?
> >
> > Thank you,
> > Brian
> >
> >
> >
> >
> >
> > Date: Mon, 1 Sep 2014 21:34:58 +0200
> > From: Gaël Thomas <gael.thomas00 at gmail.com>
> > To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>,
Nicolas
> > Geoffray <nicolas.geoffray at gmail.com>
> > Subject: [LLVMdev] VMKit is retired (but you can help if you want!)
> > Message-ID:
> > <CAOWuPDcZBpt_JJ5yo5YN=C+RWbtbneXB1UGd90d0mXdnrs8=RQ at
mail.gmail.com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > Hi all,
> >
> > So, as explained in the LLVM weekly, the VMKit project is retired. It
> > was a very fun project, but we don't have any manpower to maintain
the
> > code since one year. If someone is interested by restarting the
> > project, just send me an email, I can help to understand the
> > architecture of the project and how it works. I'm pretty sure that
we
> > can also extract a Java to LLVM compiler easily (I know that some
> > people are interested by that).
> >
> > And I want to thank all the LLVM team for their support during these
> > last ten (yes ten:)) years! Without their help, it would have been
> > impossible to develop VMKit.
> >
> > See you and thank you!
> > Gaël
> >
> >
> >
> >
>
>
>
> --
> -------------------------------------------------------------------
> Gaël Thomas, Associate Professor, UPMC
> http://pagesperso-systeme.lip6.fr/Gael.Thomas/
> -------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140910/c0d1336d/attachment.html>
Brian Faull
2014-Oct-17 15:53 UTC
[LLVMdev] VMKit is retired (but you can help if you want!)
Hello again Gaël, et al -
I have returned to attempt modifications to "fix" VMKit to produce a
working Java AOT! I would like to complete the modifications to VMKit AOT, and
I would sincerely appreciate your help to keep me moving in the right direction.
From your email on- and off-list, I understand that you have successfully
reproduced the error I reported, and you checked in a VMKit branch
("http://llvm.org/svn/llvm-project/vmkit/branches/aot" ) with some
additions and modifications -- thank you! I checked out your changes, but I
have some questions.
1- Build / code changes
1A- Using examples/aot/Makefile to build target aot-test
The build rule in this Makefile for "%.class.o : %.class.bc" isn't
used when building intermediate target
vmkit_aot/examples/aot/Release+Asserts/HelloWorld.class.o, and so it doesn't
get additional flags ("-load
/path/to/vmkit_aot/Release+Asserts/lib/static-gc-printer.so -j3-aot"). The
generic rule "%.o : %.bc" from the included ../../Makefile.rules is
used. I don't exactly know why; I thought pattern rules favored shorter
stem. I moved the rule to %.class.bc.o:%.class.bc to *before* the `include` to
imply preference, and it works. Certainly there are other ways to solve this.
The same is true for examples/aot/java/Makefile .
1B- With that hack, when linking aot-test, I get the same "multiple
definition of" the following symbols:
__L__Vstatic_buf
__L__Vvirtual_buf
____Vstatic_buf
____Vvirtual_buf
because they are defined both in HelloWorld.class.o (in libHelloWorld.a) and in
/path/to/vmkit_aot/Release+Asserts/lib/libPrecompiled.a . At first glance, the
disassembly of both versions of __L__Vstatic_buf are equivalent. I experimented
with stripping these symbols from one or both of the offending objects, such as
adding this line to the recipe for HelloWorld.class.o
strip --strip-symbol=__L__Vstatic_buf --strip-symbol=__L__Vvirtual_buf
--strip-symbol=____Vstatic_buf --strip-symbol=____Vvirtual_buf $@
The result of removing the symbols from either object is indistinguishable (to
me).
Can you explain these to me:
I- Where do these symbols come from?
II- What do they do?
III- How should this conflict be resolved (e.g., which version to use)?
1C- With that hack, I can build vmkit_aot/Release+Asserts/bin/aot-test (default
target of examples/aot/Makefile), but it fails with the same "Wrong stack
trace" error as before:
aot-test: JavaRuntimeJIT.cpp:381: void *j3ResolveVirtualStub(j3::JavaObject
*): Assertion `FI->Metadata != __null && "Wrong stack
trace"' failed.
I assume this is "the last" error. :) I can step through in the
debugger, but I don't understand what's wrong and/or what needs to be
changed.
2- Documentation
In previous emails, you had said ... :
2A- you understand the problem with VMKit AOT. Can you describe your
understanding of this problem and how to fix it? Any level of detail would be
helpful.
2B- you added a TODO list, but I don't see it in the repository. Would you
show me where it is? This TODO list would be very helpful for a VMKit hacker
like me :)
2C- you were working on documentation of the VMKit design. I don't mean to
pester you, but this would also be very helpful for me when learning how to make
any necessary changes to VMKit.
Thank you again,
Brian
On Sep 10, 2014, at 2:24 PM, Brian Faull <bfaull at cog-e.com> wrote:
> Okay -- thank you for the suggestions, but without my hack (which I will
happily abandon! :) I'm still getting the error "Should have found a
JavaMethod". I'll describe what I'm doing and attach the bitcode
as you requested.
>
> Here's what I'm doing, if you're able to recreate:
> * installed OpenJDK 6u23
> * clean version of VMKit from the repository, configured as follows:
> ./configure --with-llvm-config-path=/path/to/llvm33/bin/llvm-config
--with-classpath-impl=openjdk --with-openjdk-path=/path/to/jdk1.6.0_23
> where directory /path/to/jdk1.6.0_23 contains the installation of
OpenJDK6u23, e.g., bin, jre, lib, include.
> * your class-to-bitcode commands, which includes several things I
hadn't tried. Specifically, I'm invoking as follows:
> vmjc -print-aot-stats HelloWorld.class
> opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass -o
HelloWorld.class.gc.bc HelloWorld.class.bc
> llc -load=../../Release+Asserts/lib/static-gc-printer.so
HelloWorld.class.gc.bc
>
> The last command fails with the "Should have found a JavaMethod"
error. The only difference from yours is the order of arguments to `opt`, just
to strictly follow the `opt [options] <input bitcode file>` usage.
>
> I think the `vmjc` command needs additionally the "-main="
argument, but I have tried both ways with the same outcome.
>
> Further comments inline below, much snipping.
>
>
> On Sep 9, 2014, at 7:42 PM, Gaël Thomas <gael.thomas00 at gmail.com>
wrote:
>> Oups, sorry for the mistake, llcj (not llc:)) is not more maintained!
>>
>> Gaël
>>
>> Le 10 sept. 2014 00:27, "Gaël Thomas" <gael.thomas00 at
gmail.com> a écrit :
>> Hi Brian,
>>
> ...
>>
>> So, I answer in your previous mail. As it's the beginning of the
year
>> (I'm teaching), I haven't found any time to install vmkit and
make the
>> tests. But we will proceed step by step to understand the problem. In
>> parallel, I'm also writing a documentation to describe the
internals
>> of VMKit, it will be useful for you and other people interested by the
>> AOT.
>
> Fantastic; such a document would be very helpful; thank you. And good luck
with the teaching!
>
>> You only have to use openjdk (gnu classpath is not more maintained as
>> the project is also retired:)). We have a bug with the new versions of
>> openjdk, and we are using the 6u23. I don't know if all the
previous
>> versions will be ok. So, I suggest using this version.
>
> I found OpenJDK6u23 somewhat difficult to find; for others searching
it's at
>
http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase6-419409.html#jdk-6u23-oth-JPR
>
> I had previously not found a version this old at
> https://java.net/projects/openjdk6/downloads
>
>>
>> > => The output I see is as follows:
>> > =========>> > llc: VmkitGCPrinter.cpp:268: llvm::Constant
*FindMetadata(const llvm::Function &): Assertion `0 && "Should
have found a JavaMethod"' failed.
> ...
>> > Are we on the right path? Is this a good start?
>>
>> It's a very good start :) But your hack is not the good solution
>> because you should return from FindMetaData at the line 229 (you
>> should not reach the test at line 239).
>
> Good start! This seems to be the problem, because I seem to be in the
wrong code path.
>
>>
>> So, can you try this compilation scheme (without your hack in
>> StaticGCPrinter.cpp:))?
>>
>> vmjc -print-aot-stats HelloWorld.class
>> opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass
>> HelloWorld.class.bc -o HelloWorld.class.gc.bc
>> llc -load=../../Release+Asserts/lib/static-gc-printer.so
HelloWorld.class.gc.bc
>
> With the commands as you suggest, I step through the execution of the `llc`
command in the debugger. When the HelloWorld::main() function is passed to
FindMetadata(), the F.getName() is
"JnJVM_HelloWorld_main___3Ljava_lang_String_2" and F.UseList is NULL
(wrong, I think). This forces the code path to drop immediately out of the
first for() loop (past the return at VmkitGCPrinter.cpp:229, and on to the block
at :239). In contrast, when the <init> function (F.getName() is
"JnJVM_HelloWorld__0003Cinit_0003E__") is processed in FindMetadata(),
F.UseList is non-NULL.
>
>>
>> If it does not change anything, can you send me the
>> HelloWorld.class.bc file? I will quickly see if everything is ok for
>> the GC and where is the problem.
>
> The bitcode is (effectively) identical before and after the `opt` line,
which is presumably not correct. `llvm-diff` reports no differences, and a
`diff` of the `llvm-dis` output reports only that the module names are
different. I will attach the file off-list because I expect it will get
scrubbed anyway.
>
> The above is with your exact commands. Adding the
"-main=HelloWorld" argument to `vmjc` puts 3 additional symbols in the
bitcode:
> U StartJnjvmWithoutJIT
> T main
> d mainClass
> That is, adds a main() in addition to the HelloWorld::main() (which gets
mangled as "JnJVM_HelloWorld_main___3Ljava_lang_String_2"). I think
this is necessary and correct for non-JIT linking and execution, but that's
perhaps further down the line.
>
> I am continuing to look at the situation, but I would appreciate hearing
anything that you can see from the bitcode file.
>
> Thank you again!
> Brian
>
>
>>
>>
>> 2014-09-09 21:51 GMT+02:00 Brian Faull <bfaull at cog-e.com>:
>> > Hello again Gaël, (et al)
>> >
>> > More on rekindling work on VMKit! Thank you for your interactions
thus far on- and off-list.
>> >
>> > As you suggested in your VMKit-retirement email (to which I'm
attempting to respond), I'm interested in producing a Java-to-LLVM compiler
out of VMKit. I'd like to take you up on your offer to help understand the
architecture. If I can get the a Java-to-LLVM compiler working for my purposes,
I'll be happy to maintain at least this part of the VMKit project.
>> >
>> > To that end, I have exactly one fundamental question to start:
>> >
>> >
>> > How do you suggest to perform Java-to-LLVM compilation using
VMKit?
>> >
>> >
>> > Here are my thoughts on this:
>> >
>> > It looks like the `llcj` tool (VMKit/tools/llcj/llcj.cpp) was
meant for this; it declares itself as "Java to native compiler" but it
appears that it hasn't been maintained (e.g., deprecated interfaces,
no-longer-existent command-line options and libraries). `llcj` is described in
Appendix A of Geoffray's thesis,
http://pagesperso-systeme.lip6.fr/Nicolas.Geoffray/these-geoffray.pdf linked
from the VMKit site.
>> >
>> > `llcj` attempts the following to translate Java to native:
>> > * vmjc (Java .class => LLVM bitcode .bc)
>> > * opt (LLVM bitcode optimizer)
>> > * llc (LLVM bitcode => target assembly .s)
>> > * gcc (assemble and link)
>> >
>> > Is that "The Right Way"?
>> >
>> > After some time hacking various changes, I can use basically that
method to compile VMKit/tools/trainer/HelloWorld.java into a linked executable
(but it barfs at runtime and I can't fix it).
>> >
>> > I'm doing roughly the following, on x86_64, using LLVM3.3,
OpenJDK 1.6.0 build 30, a debug build of VMKit (required small build-hacks,
which I could describe on request), and the (perhaps-incorrect) edit to
`static-gc-printer.so` described in an earlier post:
http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/076128.html :
>> > cd /path/to/vmkit/tools/trainer
>> > javac HelloWorld.java
>> > ../../Debug+Asserts/bin/vmjc -main=HelloWorld -print-aot-stats
HelloWorld.class
>> > /path/to/llvm33/bin/llc
-load=../../Debug+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
>> >
>> > That produces for me a reasonable native assembly file.
>> >
>> > Next I assemble and link, using a set of libraries/objects
inspired by `llcj` and refined by a semi-automated trial-and-error:
>> > /path/to/llvm33/bin/clang++ -g3 -O0 -o HelloWorld.class.exe
HelloWorld.class.sed.s -L/path/to/vmkit_test-codechanges/Debug+Asserts/lib
-L/path/to/llvm33/lib -pthread -lm -ldl -lz -lJ3 -lClasspath -lJ3Compiler
-lCommonThread -lVmkit -lVmkitCompiler -lPrecompiled -lFinalMMTk
-lLLVMX86Info -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Utils -lLLVMX86AsmPrinter
-lLLVMExecutionEngine -lLLVMScalarOpts -lLLVMipo -lLLVMipa -lLLVMAnalysis
-lLLVMInstCombine -lLLVMInstrumentation -lLLVMTarget -lLLVMJIT -lLLVMRuntimeDyld
-lLLVMObject -lLLVMObjCARCOpts -lLLVMVectorize -lLLVMTransformUtils
-lLLVMSelectionDAG -lLLVMCodeGen -lLLVMMC -lLLVMCore -lLLVMSupport -rdynamic
>> >
>> > This links and executes, but here are my problems/hacks:
>> > * `llc` generates an assembly file with symbols (e.g.,
____Vstatic_buf() and friends) that conflict with one of the archives listed
above (libPrecompiled.a, which I believe is necessary for java_lang_Object and
others). I basically remove those symbols (sed '/_buf$/ s/globl/weak/'
HelloWorld.class.s > HelloWorld.class.sed.s) before assembling. I don't
know if this is a reasonable hack or if it's causing problems.
>> > * I get the following error at runtime (because some fields are
NULL) and I can't make progress:
>> > HelloWorld.class.exe: JavaRuntimeJIT.cpp:381: void
*j3ResolveVirtualStub(j3::JavaObject *): Assertion `FI->Metadata != __null
&& "Wrong stack trace"' failed.
>> >
>> > Am I on the right path, or should I be considering other methods?
Or wrong entirely?
>> >
>> > Ultimately, I propose changing `llcj` into something like a Python
script that executes the appropriate utilities to start from Java source and
resulting in a linked target-architecture executable, using `clang++` as
compiler-driver for assembling and linking. It would be essentially a work-alike
to `gcj`.
>> >
>> > Can you help point me in the right direction?
>> >
>> > Thank you,
>> > Brian
>> >
>> >
>> >
>> >
>> >
>> > Date: Mon, 1 Sep 2014 21:34:58 +0200
>> > From: Gaël Thomas <gael.thomas00 at gmail.com>
>> > To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>,
Nicolas
>> > Geoffray <nicolas.geoffray at gmail.com>
>> > Subject: [LLVMdev] VMKit is retired (but you can help if you
want!)
>> > Message-ID:
>> > <CAOWuPDcZBpt_JJ5yo5YN=C+RWbtbneXB1UGd90d0mXdnrs8=RQ at
mail.gmail.com>
>> > Content-Type: text/plain; charset=UTF-8
>> >
>> > Hi all,
>> >
>> > So, as explained in the LLVM weekly, the VMKit project is retired.
It
>> > was a very fun project, but we don't have any manpower to
maintain the
>> > code since one year. If someone is interested by restarting the
>> > project, just send me an email, I can help to understand the
>> > architecture of the project and how it works. I'm pretty sure
that we
>> > can also extract a Java to LLVM compiler easily (I know that some
>> > people are interested by that).
>> >
>> > And I want to thank all the LLVM team for their support during
these
>> > last ten (yes ten:)) years! Without their help, it would have been
>> > impossible to develop VMKit.
>> >
>> > See you and thank you!
>> > Gaël
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> -------------------------------------------------------------------
>> Gaël Thomas, Associate Professor, UPMC
>> http://pagesperso-systeme.lip6.fr/Gael.Thomas/
>> -------------------------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141017/8a445c14/attachment.html>