llvm dev - Feb 2014 - [LLVMdev] [GSoC 2014] Using LLVM as a code-generation backend for Valgrind

Hi,

I've seen on the LLVM's Open Projet Page [1] an idea about using LLVM to
generate native code in Valgrind. For what I know, Valgrind uses libVEX 
to translate native instructions into a bitcode, used to add the 
instrumentation and then translated back to native code for execution.

Valgrind and LLVM are two tools that I use nearly every day. I'm also 
very interested in code generation and optimization, so adding the 
possibility to use LLVM to generate native code in libVEX interests me 
very much. Is it a good idea? Could a LLVM backend bring something 
useful to Valgrind (for instance, faster execution or more targets 
supported) ?

I've sent this message to the LLVM and Valgrind mailing lists because 
I've originally found the idea on the LLVM's website, but Valgrind is 
the object of the idea. By the way, does anyone already know if LLVM or 
Valgrind will be a mentoring organization for this year's GSoC?

You can find in [2] the list of my past projects. During the GSoC 2011, 
I had the chance to use the Clang libraries to compile C code, and the 
LLVM JIT to execute it (with instrumented stdlib functions). I have also 
played with the LLVM C bindings to generate code when I explored some 
parts of Mesa.

Denis Steckelmacher

[1] : http://llvm.org/OpenProjects.html#misc_new
[2] : http://steckdenis.be/page-projects.html

I work on LLVM and used to work on Dr. Memory, a memory debugger similar to
Valgrind's Memcheck.  I think using LLVM as a Valgrind backend is probably
feasible, but I highly doubt that LLVM will be able to generate code
quickly enough to make it usable.  It might be worthwhile as a second level
JIT, but usually the major downside to DBI tools is the startup overhead,
and LLVM code generation can only increase that.

That doesn't mean it wouldn't be a fun project, though.  :)


On Mon, Feb 24, 2014 at 10:51 AM, Denis Steckelmacher
<steckdenis at yahoo.fr>wrote:
> Hi,
>
> I've seen on the LLVM's Open Projet Page [1] an idea about using
LLVM to
> generate native code in Valgrind. For what I know, Valgrind uses libVEX to
> translate native instructions into a bitcode, used to add the
> instrumentation and then translated back to native code for execution.
>
> Valgrind and LLVM are two tools that I use nearly every day. I'm also
very
> interested in code generation and optimization, so adding the possibility
> to use LLVM to generate native code in libVEX interests me very much. Is it
> a good idea? Could a LLVM backend bring something useful to Valgrind (for
> instance, faster execution or more targets supported) ?
>
> I've sent this message to the LLVM and Valgrind mailing lists because
I've
> originally found the idea on the LLVM's website, but Valgrind is the
object
> of the idea. By the way, does anyone already know if LLVM or Valgrind will
> be a mentoring organization for this year's GSoC?
>
> You can find in [2] the list of my past projects. During the GSoC 2011, I
> had the chance to use the Clang libraries to compile C code, and the LLVM
> JIT to execute it (with instrumented stdlib functions). I have also played
> with the LLVM C bindings to generate code when I explored some parts of
> Mesa.
>
> Denis Steckelmacher
>
> [1] : http://llvm.org/OpenProjects.html#misc_new
> [2] : http://steckdenis.be/page-projects.html
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140224/3c90b311/attachment.html>

On 25 Feb 2014, at 07:50, Reid Kleckner <rnk at google.com> wrote:
> I work on LLVM and used to work on Dr. Memory, a memory debugger similar to
Valgrind's Memcheck.  I think using LLVM as a Valgrind backend is probably
feasible, but I highly doubt that LLVM will be able to generate code quickly
enough to make it usable.  It might be worthwhile as a second level JIT, but
usually the major downside to DBI tools is the startup overhead, and LLVM code
generation can only increase that.
It would also be interesting to cache the LLVM-generated code between runs, or
even do it as an AOT pass.  A lot of the times I have problems where Valgrind
would help, I end up needing multiple runs and each one takes several minutes of
execution time (sometimes 5-10, because of the slowdown).  I've also found
that for some applications, Valgrind's overhead makes it impossible to
reproduce some timing issues and so the segfault just goes away if you run the
code slower.

David

On 2/24/14 12:51 PM, Denis Steckelmacher wrote:
> Hi,
>
> I've seen on the LLVM's Open Projet Page [1] an idea about using
LLVM
> to generate native code in Valgrind. For what I know, Valgrind uses 
> libVEX to translate native instructions into a bitcode, used to add 
> the instrumentation and then translated back to native code for 
> execution.
I think a more interesting idea would be to use LLVM to perform 
instrumentation and then to use Valgrind to instrument third-party 
libraries linked into the program.

What I'm imagining is this: Let's say you instrument a program with 
SAFECode or Asan to find memory safety errors.  When you run the program 
under Valgrind, the portion of the code instrumented by SAFECode or Asan 
runs natively without dynamic binary instrumentation because it's 
already been instrumented.  When the program calls uninstrumented code 
(e.g., code in a dynamic library), Valgrind starts dynamic binary 
instrumentation to do instrumentation.

A really neat thing you could do with this is to share run-time data 
structures between the LLVM and Valgrind instrumentation.  For example, 
Valgrind could use SAFECode's meta-data on object allocations and 
vice-versa.

If you're really clever, the LLVM instrumentation could be added in a 
way where it's off by default by enabled when the program is run under 
Valgrind.

The net effect is that most of the instrumentation works faster because 
it was added at compile-time, but code compiled with another compiler 
can still be instrumented by Valgrind with a performance penalty.

-- John T.

-valgrind-dev, it bounced for me
+timurrrr
+zhaoqin
+eugenis

On Tue, Feb 25, 2014 at 7:50 AM, John Criswell <criswell at
illinois.edu>wrote:
> On 2/24/14 12:51 PM, Denis Steckelmacher wrote:
>
>> Hi,
>>
>> I've seen on the LLVM's Open Projet Page [1] an idea about
using LLVM to
>> generate native code in Valgrind. For what I know, Valgrind uses libVEX
to
>> translate native instructions into a bitcode, used to add the
>> instrumentation and then translated back to native code for execution.
>>
>
> I think a more interesting idea would be to use LLVM to perform
> instrumentation and then to use Valgrind to instrument third-party
> libraries linked into the program.
>
We did this with DynamoRIO, ASan, and MSan, and published the results:
http://research.google.com/pubs/pub41440.html

It's a cool idea, but we haven't been able to productionize it enough to
test Chromium yet.  The code for the msan side is actually in compiler-rt:
http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/msandr/msandr.cc?view=markup

Ultimately it may be easier (on Linux) to build new instrumented packages
for every library that you care about testing with.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140225/a644533b/attachment.html>

On 02/25/2014 04:50 PM, John Criswell wrote:
>
> I think a more interesting idea would be to use LLVM to perform
> instrumentation and then to use Valgrind to instrument third-party
> libraries linked into the program.
>
> What I'm imagining is this: Let's say you instrument a program with
> SAFECode or Asan to find memory safety errors.  When you run the program
> under Valgrind, the portion of the code instrumented by SAFECode or Asan
> runs natively without dynamic binary instrumentation because it's
> already been instrumented.  When the program calls uninstrumented code
> (e.g., code in a dynamic library), Valgrind starts dynamic binary
> instrumentation to do instrumentation.
>
> A really neat thing you could do with this is to share run-time data
> structures between the LLVM and Valgrind instrumentation.  For example,
> Valgrind could use SAFECode's meta-data on object allocations and
> vice-versa.
>
Someone proposed to cache the results of a JIT compilation. Caching LLVM 
bitcode is easy (and the LLVM optimizations operate on bitcode, so they 
don't need to be re-run on bitcode reload), and may be a good way to 
fasten Valgrind. Caching native binary code is more difficult and would 
only be useful if LLVM's codegen is slow (I think that the codegen can 
be configured to be fast, for instance by using a simpler register 
allocator).

If every .so is cached in a separate bitcode file, loading an 
application would only require the generation of bitcode for the 
application itself, not the libraries it uses, provided that they didn't 
change since another application using them was analyzed. That may speed 
up the start-up of Valgrind.

Hi,

I did some work in libVEX a few years ago.
AFAIR, the gains from improved code generation in libVEX are very hard to be 
noticeable, and they may even increase the overall run-time since the 
performance improvement in the code quality may not payoff the extra time 
you spend in analysis (well, that's probably true for every JIT compiler).
Therefore, and since an LLVM JIT would significantly increase the run-time 
of libVEX's code gen, I think this probably only makes sense for hot super 
basic blocks. Most BBs are irrelevant. libVEX doesn't have a profiling 
infrastructure to pin-point hot BBs, AFAIR, though.
One of valgrind's weaknesses is SIMD code. For example, openssl code under 
valgrind takes forever to execute.  LLVM could potentially help there since 
it now has (two) BB vectorizers.

I'm sure the (very friendly) Valgrind developers can give you a few more 
hints (and correct me if I'm wrong).
I'm clueless about this year's GSoC.

Nuno

----- Original Message ----- 
From: "Denis Steckelmacher" <steckdenis at yahoo.fr>
To: <valgrind-developers at lists.sourceforge.net>; <llvmdev at
cs.uiuc.edu>
Sent: Monday, February 24, 2014 6:51 PM
Subject: [LLVMdev] [GSoC 2014] Using LLVM as a code-generation backend 
forValgrind

> Hi,
>
> I've seen on the LLVM's Open Projet Page [1] an idea about using
LLVM to
> generate native code in Valgrind. For what I know, Valgrind uses libVEX to 
> translate native instructions into a bitcode, used to add the 
> instrumentation and then translated back to native code for execution.
>
> Valgrind and LLVM are two tools that I use nearly every day. I'm also
very
> interested in code generation and optimization, so adding the possibility 
> to use LLVM to generate native code in libVEX interests me very much. Is 
> it a good idea? Could a LLVM backend bring something useful to Valgrind 
> (for instance, faster execution or more targets supported) ?
>
> I've sent this message to the LLVM and Valgrind mailing lists because
I've
> originally found the idea on the LLVM's website, but Valgrind is the 
> object of the idea. By the way, does anyone already know if LLVM or 
> Valgrind will be a mentoring organization for this year's GSoC?
>
> You can find in [2] the list of my past projects. During the GSoC 2011, I 
> had the chance to use the Clang libraries to compile C code, and the LLVM 
> JIT to execute it (with instrumented stdlib functions). I have also played 
> with the LLVM C bindings to generate code when I explored some parts of 
> Mesa.
>
> Denis Steckelmacher
>
> [1] : http://llvm.org/OpenProjects.html#misc_new
> [2] : http://steckdenis.be/page-projects.html
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev