llvm dev - Oct 2007 - [LLVMdev] OT: new here, dynamic/runtime compilation (in general)

well, sadly, I am not sure how people are on this list...
in any case, LLVM is an interesting project, and may well continue being
interesting.


but, in my case, I have done my own compilation framework...

ok, I didn't really hear about the really interesting bits of LLVM until
after I had (more or less) wrote mine...

ok, my point is to maybe to have something interesting to talk about, not as
much to try to spam my own effort (so, I hope that people here do not take
offense).


in a more or less completely custom written compiler framework (I am the only
programmer in my case, a lone hobbyist), this project originally starting about
march or so (though some parts are older, and were reused as such).

far from being all that complete or featureful, and currently my version only
has a working x86 version, and an incomplete x86-64 version...

as of yet, it is still often broken, and sometimes buggy (when I find bugs, I
fix then, but I have hardly been doing all that comprehensive of testing, and as
of yet, I am paranoid of the horrible levels of breaking such testing would
reveal...).


now, what I do with it is this:
I use C as a scripting language...

ok, C is also my main programming language, as I very rarely use C++ for much of
anything...
(something like embedded-C++ may also be possible at some point, but full C++ is
unlikely as it just looks too painful to write a compiler for...).


so, I load the C modules dynamically, and have them link directly to the host
app (on windows, this requires manually opening and processing the exe's
contents, but on linux, I have libdl...).

the reason is this:
what is better than C at interfacing with an otherwise C codebase?...
if I have no need of wrappers or FFI crap (needed for nearly any other non-C
language), all the better (this was another major design goal).

for all this, I also needed a compiler (not an interpreter...), and also
something that is if possible purely dynamic.

my idea was that we go from good old static compilation and linking, to
something far more dynamic (imagine, for example, if we could use the compiler
in much the same way we use 'malloc', and piece together code much like
we currently piece together memory objects, though in a more metaphoric than
literal sense...).


for example, it was a design goal that it be possible to incrementally replace
parts of the running app image (at least, partly, aka, in VM controlled parts of
the app), for example, by redefining some already running function at runtime,
... and have other code automatically go over to using the new version (thus
requiring dynamic re-linking on the part of the framework...).

another minor goal:
fully dynamic piecewise compilation (like for example, a lisp style
'eval') has not yet been achieved (a major hurtle here is C's
semantics, meaning that nearly any language which could work in an eval-style
manner is necessarily no longer really C, which is sad...).

so, at present, it only handles source modules (with include files and a
toplevel and so on), but oh well...


C conformance:
well mostly it implements an as of yet incomplete version of C. some features
are still lacking, but they are features that are sufficiently infrequent that I
can make due without them for the time being (namely: 'static',
initialized structs, multi-prototypes, C99 style dynamicly-sized arrays, ...).

to 'add insult to injury' though, I implemented a few compiler extension
features:
some partly derived from gcc (__real, __imag, ...);
some custom: builtin geometric vectors and quaternions (I do a lot of 3D
stuff...);
..

I also have Garbage Collection, Dynamic Typing, and Prototype OO available as
library features (note, I mean 'Prototype OO', in reference to the
object system used in Self, and crudely immitated in JavaScript, and not in
reference to "some crude mockup of OOP in C"...).

well, one can use these and pretend they were using a real script language...
note that these library features do not depend on compiler extensions, and so
they also still work in parts of my project compiled with gcc...

though technically possible, I am a little more uncertain about adding compiler
extensions for stuff this much unlike C...

well, all this works for me...


performance:
well, this was never really a major goal of mine (as long as it was tolerable I
guess), but it seems oddly enough to generally produce better code than GCC,
which is probably worth something...

I don't set out to do elaborate tricks to gain performance. instead most of
what I have was gained by tweaking code produced in the "common, special
case" (looking at assembler, "well, this stupid-looking construction
is appearing far too often, may go and fix it").


intermediate language:
at this point, my project and LLVM are somewhat different.

LLVM using its good old variable and three-address-code based model.

mine works quite a bit different.
my IL is technically, a little closer to something like postscript or forth (not
very nice looking, but hell, it is produced by a compiler...).

note that it is, oddly enough, passed between the compiler stages in a texual
form, which was done in an attempt to resolve some major technical differences
between the mechanics of my upper and lower compilers.


it represents an abstract stack machine (though, mind you, this may have little
to do with the stack as seen by the HW or at the assembler level).

unlike forth, since this stack is more an abstraction than a representation of
the physical stack, it is not valid to attempt to use conditionals in any way
that would change the ordering or structure of the stack elements. long
arguments came up over these points, that I am dealing with a declarative rather
than an imperative notion of a stack...

as a result, in a few cases I have what basically amount to phi operators for
the stack, which are used for dealing with these issues (with the restriction
that all the values merging into this operator have to have the same type...).


likewise, unlike an 'actual' stack, there is no real notion of a stack
element size (for example, pointers, integers, long longs, vectors, and structs,
can all be pushed to the stack and treated as single items regardless of
worrying about type and size).

likewise, literals and symbols (names referring to variables, functions, or
types) can also go on the stack.

though also statically typed (at runtime), most operations are fairly
type-agnostic (it is the task of the operation itself to deal with whatever was
given to it).

as such, a lot of the IL-compiler's internals go to dealing with all the
various types (a good deal exists in terms of elaborate type-dispatch code).

I also used a stack because that is what I am most fammiliar with...


I call my IL language RPNIL, or RIL (since it is an RPN-based IL).
the RIL compiler (converts RIL to assembler) is generally referred to as the
'lower compiler', with the 'upper compiler' handling the
translation from C to RIL.

as noted, I also have a runtime assembler and linker...


or such...

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20071022/6759e27b/attachment.html>

On Oct 21, 2007, at 5:27 PM, BGB wrote:
> well, sadly, I am not sure how people are on this list...
I'd suggest browsing through the llvmdev mailing list archive.
> now, what I do with it is this:
> I use C as a scripting language...
>
Sounds like a fun project.  Note that the clang C front-end also  
supports use in a JIT environment: this will allow you to JIT the  
full generality of C code (when clang is complete).  Today while  
clang is not complete, it is far enough along to be very useful for a  
wide variety of numeric environments, as its scalar, vector, and  
array support are quite advanced.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20071021/26449460/attachment.html>

----- Original Message ----- 
  From: Chris Lattner 
  To: LLVM Developers Mailing List 
  Cc: BGB 
  Sent: Monday, October 22, 2007 10:44 AM
  Subject: Re: [LLVMdev] OT: new here, dynamic/runtime compilation (in general)




  On Oct 21, 2007, at 5:27 PM, BGB wrote:


    well, sadly, I am not sure how people are on this list...


  I'd suggest browsing through the llvmdev mailing list archive.

   
yes, may make sense.
just, I encounter a lot of places where people start flaming as soon as the
topic drifts at all OT (as per their personal definitions).



    now, what I do with it is this:
    I use C as a scripting language...




  Sounds like a fun project.  Note that the clang C front-end also supports use
in a JIT environment: this will allow you to JIT the full generality of C code
(when clang is complete).  Today while clang is not complete, it is far enough
along to be very useful for a wide variety of numeric environments, as its
scalar, vector, and array support are quite advanced.

yes.
clang is interesting, so I will continue to watch and hope it does something
cool.

sorry for not having done much research here (if it so happens all this exists
already), but I will assume that it likely implements the same basic kind of
vector handling as gcc (rather than geometric vectors). they need not be
confused (per-element multiplication is an almost useless operation in geometry,
but dot-product is almost mandatory...).

ok, just in my case, I guess I am a little fussy/weird, wanting efficient dot
and cross products  (if at all possible, though these operations at present
don't exist natively within SSE...), among many other operations (length,
lerp, renormalization, ...).

I suspect the people behind gcc for whatever reason did not know of or
couldn't really see much need for geometric vector support...


I also use quaternions (as yet another tweaky compiler extension, basically,
they look like 4 element float vectors but have a few different operations from
those of 4-element geometric vectors).

quaternions are also a complex space, and can thus also be used as part of the
normal numeric tower (much as good old complexes...). though a little arbitrary,
I also allow operations between complexes and quaterions (the complex
automatically gets promoted to a quat in the process), but it is arbitrary in
that the complexes are arbitrarily mapped to the WI plane (much as how 2D
vectors automatically map to the XY plane in 3D situations, or converting a
3D-vector to a 2D vector often involves simply discarding Z, even though the XY
in a 2D vector could concievably map to any possible plane within a 3D
space...).

it is arbitrary, but it works...


but, I guess I will have to investigate all this (to see more what LLVM does and
does not support...).

dunno if, or if there would be any intention to implement features like these (I
would not mind that much though if they were only builtin functions or similar,
as that is how my gcc-fallback support works, but in the case of my compiler,
many of these operations also exist via overloaded operators).

then again, probably a fairly smart compiler could still get plenty good results
out of inline functions and intrinsics or something (as it so happens, my
compiler is not so smart here, so I made these features builtin, and fell back
to functions for more involved operations, such as quaternion slerp, ...).

or something...




  -Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20071022/1630d007/attachment.html>

On Oct 21, 2007, at 6:23 PM, BGB wrote:
> ok, just in my case, I guess I am a little fussy/weird, wanting  
> efficient dot and cross products  (if at all possible, though these  
> operations at present don't exist natively within SSE...), among  
> many other operations (length, lerp, renormalization, ..).
>
> then again, probably a fairly smart compiler could still get plenty  
> good results out of inline functions and intrinsics or something  
> (as it so happens, my compiler is not so smart here, so I made  
> these features builtin, and fell back to functions for more  
> involved operations, such as quaternion slerp, ...).
>
llvm-gcc and clang both fully support GCC style generic vectors and  
altivec/sse intrinsics.  In addition, clang supports "glsl" style  
vector permutations, direct vector element access, etc.  The LLVM  
optimizer and code generator supports many vector operations  
independent of the front-end syntax, as well as highly optimized code  
generation for Altivec and SSE.

-Chris


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20071021/65256cc8/attachment.html>

----- Original Message ----- 
  From: Chris Lattner 
  To: BGB 
  Cc: LLVM Developers Mailing List 
  Sent: Monday, October 22, 2007 12:28 PM
  Subject: Re: [LLVMdev] OT: new here, dynamic/runtime compilation (in general)




  On Oct 21, 2007, at 6:23 PM, BGB wrote:


    ok, just in my case, I guess I am a little fussy/weird, wanting efficient
dot and cross products  (if at all possible, though these operations at present
don't exist natively within SSE...), among many other operations (length,
lerp, renormalization, ..).

    then again, probably a fairly smart compiler could still get plenty good
results out of inline functions and intrinsics or something (as it so happens,
my compiler is not so smart here, so I made these features builtin, and fell
back to functions for more involved operations, such as quaternion slerp, ...).




  llvm-gcc and clang both fully support GCC style generic vectors and
altivec/sse intrinsics.  In addition, clang supports "glsl" style
vector permutations, direct vector element access, etc.  The LLVM optimizer and
code generator supports many vector operations independent of the front-end
syntax, as well as highly optimized code generation for Altivec and SSE.

yes, cool.

in any case, people can probably build whatever they need using what is
provided...
(and, if all this is already built in, all the better...).


but, yes, it may not matter much, I simply have some stuff that is non-standard
and tweaky is all...

sorry if it seems I am obsessing, or if this topic seems inappropriate here...


looking, trying to determine if GLSL has these operations (ok, it has a few,
implemented as functions at least, like 'dot' and 'cross'). note
that my interface is a little different than GLSL, but I may change it to more
closely resemble that of GLSL.

now, of course, what I was not sure of (what I keep going on about) was not
about specific general vector features (aka: much of anything what SSE itself,
or most compilers, provide), but the specifics of geometric vectors (like in 3D
or physics).

(one has special fun trying to implement these well within the context of
SSE...).

I have some funky operator overloading as well: 'vec^vec' gives a dot
product ('vec*vec' gives a per-component product, as in gcc); and
'vec%vec' gives a cross product. for quaternions, 'quat*quat' is
overloaded for a quaternion product (grassman).

per-component access:
I treat vectors like some kind of implicit struct.
components are: x,y,z,w. i, j, and k are analogous to x, y, and z.
looks like GLSL also has r, g, b, and a as synonyms as well (don't have
these in my case, then again, I don't usually do color calculations
either...).

so, for a vector v: v.x, v.y, v.z, or v.w; or a quat q: q.i, q.j, q.k, or q.w.
I also do this for complexes, where complexes are viewed as having 2 components:
i and w (matching them with quaternions as such).

so, it is valid to type: "(float _Complex)QUAT(3,1,0,0);", which comes
out as the complex
"3+1i" (j and k are simply discarded). note that, at present, it is
not possible to type out quaternions like complexes, say
"(3+1i+0j+0k)*(0+5i+1j-2k)", though this could be a sane feature, it
would also give pressure to allow similar for vectors, such as
("3x+4y-5z", which may look odd and misleading), and simply allowing
it for complexes may be enough...

hmm: as for vectors, the funky syntax could be very useful for translation and
accelaration ("vel-=gravity*1z;", as opposed to
"gavity-=VEC3(0,0,gravity);").

who knows though...


misc, quaternions (general concept):
http://en.wikipedia.org/wiki/Quaternion




  -Chris



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20071022/fe2849b2/attachment.html>