llvm dev - Jan 2004 - [LLVMdev] Re: idea 10

Hello again Valery,

Valery A.Khamenya wrote:
> All benefits, what one could obtain from "LLVM supporting multiple CPU
> at single host", one might obtaine from "LLVM supporting multiple
CPU
> at multiple hosts". Isn't that logical?
I see more precisely what you mean, but I don't think it is that 
straightforward to generalise the benefits multiple CPU on single host 
programming to multiple CPU at multiple hosts. I don't think that both 
cases involve the same techniques.

For example, in "single host" configuration you get a very low cost
for
communicating data because you use memory instead of network. Memory is 
a low-level primitive medium for communicating data, while network is 
not a primitive medium because it involves many communication protocols.

Memory is simple to manage when it is on a single hardware, but when it 
turns to a "shared" (or "distributed") memory, things can
get pretty
complicated (because you have -among others- security, latency, 
integrity and fault-tolerance to take into account).

Moreover, there are many, many ways to implement "distributed" or 
"parallel" computing. Some approaches are based on distributing data 
according to the available CPU resources, others on distributing the 
program control flow according to the proximity of data, and so on.

What would you consider as the core primitives of single host, multi-CPU 
programming ?

Cheers,

-- Se'bastien

Just a few notes from the peanut gallery... note that I can't really say
what is and is not possible with LLVM, I can just talk about our plans
(which we believe to be possible).  If you guys are really interested in
this stuff, try hacking something together and see if it works!  :)


On Thu, 8 Jan 2004, [koi8-r] "Valery A.Khamenya[koi8-r] " 
wrote:
>
> Indeed, LLVM will allocate registers and regular memory for us. This
> output is free of any strategy of this allocation. We do *not* define
> strategy of this allocation and we are even quite happy with that!
> Therefore I could expect, that allocation of CPU could be also a
> internal back-end problem of the LLVM like registers.
In theory that is possible, but it is somewhat problematic, at least
is the current representation.  LLVM is designed to provide a convenient
level of abstraction that is good for code representation, optimization,
etc.

Though I am mostly ignorant about leading edge distributed (i.e., loosely
coupled parallel) programming stuff, I _think_ that network latencies are
such that you really need the "big picture" of what a program is doing
to
know how to profitably distribute and adapt the program to the environment
it finds itself running in.  I believe the current state of the art with
this relies on programmer annotations of various sorts to do this (Vikram
knows a lot more about this than I do, he should chime in :).  It's
possible that there is some way to expose this information down to the
LLVM level, allowing neat optimizations to happen, but I don't really
know.  :)
> In other words, indirectly, I propose to treat CPUs as the same resource
> class as memory :-) Chris, LLVMers, could you allocate CPUs resources as
> you do it for registers and memory? :P
Our mid-term plans include looking at multithreading/parallel processing
kinds of things, at least for shared memory systems.  At the LLVM level,
we are interested in _exposing_ parallelism.  In the fib example you are
using, for example, it is pretty easy to show that all of the subcalls to
fib can be run independently of each other (ie, in parallel).  Combined
with a suitable runtime library (e.g., pthreads), you could imagine an
LLVM transformation that allows it to run well on a machine with 10K
processors. :)
> Actually, maybe the problem is that scince is not ready arround
> phi-functions, SSA and so on, if we speak about calculating machine with
> several CPUs spread via network... What could LLVM-gurus say here?
"It's all about the abstraction."  :)   If you come up with a
clean design
that integrates well with LLVM, and makes sense, it's quite possible that
we can integrate it.  If it doesn't fit well with LLVM, or, well,
doesn't
make sense, then we'll probably wait for something that does.  :)  That
said, LLVM is actually a really nice platform for playing around with
"experimental" ideas like this, so feel free to start hacking once you
get a concrete idea!

-Chris

-- 
http://llvm.cs.uiuc.edu/
http://www.nondot.org/~sabre/Projects/