llvm dev - Dec 2010 - [LLVMdev] [LAD] Project proposition: llvm based dsp engine

I have been looking for a while to  LLVM as a possible technology to 
build a DSP execution engine, providing the runtime flexibility needed 
by real-time interactive DSP applications (like patcher languages), or 
by plug-in based processors, and in the same moment
the powerful link time optimizations that such a system can provide.

Such a task is daunting for a single developer project like mine, but it 
may become feasible if  such engine could be useful for multiple 
projects, so to become a community initiative (i am sending this
mail to the Linux Audio Developers mailing list and to the LLVM mailing 
list, feel free to send it elsewhere if you find it useful).

So, I'll explain in a few lines exactly what i thinking of, and i 
propose that if anybody is interested in such initiative she/he can 
contact me directly, and we will see if there is enough interest
for actually start something. Of course, i may also miss an existing 
project on the same field, in this case, let me know :->.

The idea is the following: build a tool kit, based on LLVM, to 
dynamically build the execution chain of DSP applications; almost all 
DSP applications using plug-ins have something of this kind.
This toolkit will be based on code generation and JIT, and should have a 
flexible architecture so to accomodate multiple plugin API/ABI, to be 
able to call, from the same execution chain, DSP operation written
for different contexts (for example, LADSPA plug-ins, PD objects, and so 
on). The code generation would allow this integration for a very low run 
time cost.

But the most interesting part is actually providing a definition of 
native, LLVM compiled, plug-ins, and to use the LLVM link time 
optimizations for doing things like inlining, loop fusions etc, between 
different DSP operation; this can provide quite a speed improvement for 
those chains composed of simple operators.

The execution chain should be flexible enough to provide support for 
multi-rate synchronous data flows, and should include support for 
multiple threads execution (probably different chains for different 
threads).

To complete the project, let's add a library of DSP operators, a 
reasonable high level API to use it, and documentation to help the DSP 
application designer (i find the LLVM learning curve quite steep :).

Contact maurizio at dececco.name if you are interest in setting up such a 
project; if enough developers with the right skills and enough interest 
are found I'll set up a project mailing list to further discuss the subject.

Cordially,
  Maurizio


__________
Maurizio De Cecco  -
Music:    http://www.myspace.com/mauriziodececco
Blog:     http://maurizio.dececco.name/
Software: http://www.jmax-phoenix.org/

On Mon, Dec 6, 2010 at 6:10 AM, Maurizio De Cecco <jmax at dececco.name>
wrote:
> I have been looking for a while to  LLVM as a possible technology to build
a
> DSP execution engine, providing the runtime flexibility needed by real-time
> interactive DSP applications (like patcher languages), or by plug-in based
> processors, and in the same moment
> the powerful link time optimizations that such a system can provide.
My own gut feeling on this is that you'd be better off figuring out
how to do this in the context of Faust, which already does a important
subset of what you are describing, though notably without (I think)
the LLVM part.

Note also that hosts which run plugins at the level of LADSPA/LV2,
VST, AU, DSSI etc are unlikely to be easy candidates for any
cross-plugin optimization. I can say for sure that, for example,
Ardour3 has an object called a Processor whose ::run() method
encapsulates all DSP done within Ardour, but nevertheless it would be
more or less impossible to do any kind of optimization that looked
"across" all the Processors in a signal chain (eg. gain, pan, plugins,
etc, etc).

--p

Ciao Maurizio,

2010/12/6 Maurizio De Cecco <jmax at
dececco.name>:
> I have been looking for a while to  LLVM as a possible technology to build
a
> DSP execution engine, providing the runtime flexibility needed by real-time
> interactive DSP applications (like patcher languages), or by plug-in based
> processors, and in the same moment
> the powerful link time optimizations that such a system can provide.
>
> Such a task is daunting for a single developer project like mine, but it
may
> become feasible if  such engine could be useful for multiple projects, so
to
> become a community initiative (i am sending this
> mail to the Linux Audio Developers mailing list and to the LLVM mailing
> list, feel free to send it elsewhere if you find it useful).
It's some time that I actually have a somewhat related idea: when you
use physics-based modeling techniques a crucial point is proper
scheduling of operations, which has NOT to be done on a "processing
object" (i.e., plugin or whatever) basis, but by considering
dependencies among single input/output expressions even inside
"processing objects" (this may sound unclear, I know).

This means that in practice, if you deal with compiled code (a.k.a.
plugins), you just can't reasonably do it, as opposed to specialized
tools, and in particular special audio programming languages that can
either be interpreted or compiled to some sort of "understandable"
bytecode that preserves input/output time relationships (i.e., delays)
and then run in a virtual machine that solves the scheduling issue.

An example of such specialized tools is the audio DSP language I wrote
called Permafrost. Here, the scheduling issue is solved when compiling
the Permafrost source code to LV2 plugin source (C code and Turtle/RDF
metadata), but this is a one way operation - you can't use the output
plugin code to build a new plugin taking into account these issues,
whether at runtime or not.

My idea was to define an intermediate DSP bytecode, hopefully also
capable of including metadata, and a virtual machine that schedules,
optimizes and runs the whole chain/graph.

A stupid example of what can be achieved with such a thing is the
following: suppose you have a physics-based simulator of a tube
amplifier that allows you to "plug" a physics-based loudspeaker model
into it (if you are into this kind of stuff, suppose it is WDF-based)
- you would be able to change such loudspeaker model while the system
is running and have it all correctly scheduled, optimized and
perfectly working.

On the other hand, I believe you are aiming for something different,
hence I do suggest you (as Paul did) to contact FAUST developers,
since they're pretty much into this kind of stuff, especially when it
comes to LLVM.

BR,

Stefano

Hi, and thanks for the feedback.
>My own gut feeling on this is that you'd be better off figuring out
>how to do this in the context of Faust, which already does a important
>subset of what you are describing, though notably without (I think)
>the LLVM part.
Surely a contribution to the discussion by the Faust people would
be welcome; and even more than to the discussion :->. By the way, they
do something with LLVM, but i do not know exactly what (Faust is included
in the list of projects using LLVM).

I think the interest, if any, would be in having a *low* level
library generic enough to be used by multiple applications, so
to factor the work. It may be that an existing project have internally
a large part of the needed code, though.
>Note also that hosts which run plugins at the level of LADSPA/LV2,
>VST, AU, DSSI etc are unlikely to be easy candidates for any
>cross-plugin optimization.
I was actually talking about two different context, reusing existing
binaries, and having cross plugin optimisations for native plugins.

But, if the plugins are compiled with a different compilation chain
(using the LLVM compiler producing LLVM byte code, so introducing a different
binary format for them), than cross plugin optimisation is possible also for 
existing plugins like a LADSPA plugin.

Would that be worthwhile ? Difficult to know for sure, but it is an interesting
subject to explore.

Maurizio