llvm dev - Aug 2020 - [RFC] Introduce Dump Accumulator

Introduction
===========
This RFC proposes a mechanism to dump arbitrary messages into object
files during compilation and retrieve them from the final executable.

Background
=========
We often need to collect information from all object files of
applications.  For example:

- Mircea Trofin needs to collect information from the function
  inlining pass so that he can train the machine learning model with
  the information.

- I sometimes need to dump messages from optimization passes to see
  where and how they trigger.

Now, this process becomes challenging when we build large applications
with a build system that caches and distributes compilation jobs.  If
we were to dump messages to stderr, we would have to be careful not to
interleave messages from multiple object files.  If we were to modify
a source file, we would have to flush the cache and rebuild the entire
application to collect dump messages from all relevant object files.

High Level Design
================
- LLVM: We provide machinery for individual passes to dump arbitrary
  messages into a special ELF section in a compressed manner.

- Linker: We simply concatenate the contents of the special ELF
  section.  No change is needed.

- llvm-readobj: We add an option to retrieve the contents of the
  special ELF section.

Detailed Design
==============
DumpAccumulator analysis pass
-----------------------------

We create a new analysis pass called DumpAccumulator.  We add the
analysis pass right at the beginning of the pass pipeline.  The new
analysis pass holds the dump messages throughout the pass pipeline.

If you would like to dump messages from some pass, you would obtain
the result of DumpAccumulator in the pass:

  DumpAccumulator::Result *DAR =
MAMProxy.getCachedResult<DumpAccumulator>(M);

Then dump messages:

  if (DAR) {
    DAR->Message += "Processing ";
    DAR->Message += F.getName();
    DAR->Message += "\n";
  }

AsmPrinter
----------

We dump the messages from DumpAccumulator into a section called
".llvm_dump" in a compressed manner.  Specifically, the section
contains:

- LEB128 encoding of the original size in bytes
- LEB128 encoding of the compressed size in bytes
- the message compressed by zlib::compressed

in that order.

llvm-readobj
------------

We read the .llvm_dump section.  We dump each chunk of compressed data
one after another.

Existing Implementation
======================https://reviews.llvm.org/D84473

Future Directions
================
The proposal above does not support the ThinLTO build flow.  To
support that, I am thinking about putting the message as metadata in
the IR at the prelink stage.

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200805/c7ea6c48/attachment.html>

I’m not a fan of keeping important data outside the IR in an analysis.  If we’re
planning to emit it, it should be represented directly in the IR.  Is there some
reason we can’t just stick the data in a global variable?

I’m not sure it’s helpful to have a generic mechanism for this; it’s not clear
how this would work if multiple different features were trying to emit data into
the llvm_dump section at the same time.

-Eli

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Kazu
Hirata via llvm-dev
Sent: Wednesday, August 5, 2020 3:36 PM
To: llvm-dev at lists.llvm.org; Mircea Trofin <mtrofin at google.com>; Wei
Mi <wmi at google.com>; james.henderson at sony.com
Subject: [EXT] [llvm-dev] [RFC] Introduce Dump Accumulator


Introduction

===========


This RFC proposes a mechanism to dump arbitrary messages into object

files during compilation and retrieve them from the final executable.



Background

=========


We often need to collect information from all object files of

applications.  For example:



- Mircea Trofin needs to collect information from the function

  inlining pass so that he can train the machine learning model with

  the information.



- I sometimes need to dump messages from optimization passes to see

  where and how they trigger.



Now, this process becomes challenging when we build large applications

with a build system that caches and distributes compilation jobs.  If

we were to dump messages to stderr, we would have to be careful not to

interleave messages from multiple object files.  If we were to modify

a source file, we would have to flush the cache and rebuild the entire

application to collect dump messages from all relevant object files.



High Level Design

================


- LLVM: We provide machinery for individual passes to dump arbitrary

  messages into a special ELF section in a compressed manner.



- Linker: We simply concatenate the contents of the special ELF

  section.  No change is needed.



- llvm-readobj: We add an option to retrieve the contents of the

  special ELF section.



Detailed Design

==============


DumpAccumulator analysis pass

-----------------------------



We create a new analysis pass called DumpAccumulator.  We add the

analysis pass right at the beginning of the pass pipeline.  The new

analysis pass holds the dump messages throughout the pass pipeline.



If you would like to dump messages from some pass, you would obtain

the result of DumpAccumulator in the pass:



  DumpAccumulator::Result *DAR =
MAMProxy.getCachedResult<DumpAccumulator>(M);



Then dump messages:



  if (DAR) {

    DAR->Message += "Processing ";

    DAR->Message += F.getName();

    DAR->Message += "\n";

  }



AsmPrinter

----------



We dump the messages from DumpAccumulator into a section called

".llvm_dump" in a compressed manner.  Specifically, the section

contains:



- LEB128 encoding of the original size in bytes

- LEB128 encoding of the compressed size in bytes

- the message compressed by zlib::compressed



in that order.



llvm-readobj

------------



We read the .llvm_dump section.  We dump each chunk of compressed data

one after another.



Existing Implementation

======================


https://reviews.llvm.org/D84473



Future Directions

================


The proposal above does not support the ThinLTO build flow.  To

support that, I am thinking about putting the message as metadata in

the IR at the prelink stage.



Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200805/36f4bf46/attachment.html>

I like the ability, not sure about the proposed implementation though.

Did you consider a flag that redirects `llvm::outs()` and `llvm::errs()`

into sections of the object file instead? So, you'd say:


`clang ... -mllvm -debug-only=inline ... -mllvm -dump-section=.dump`


and you'd get the regular debug output nicely ordered in the `.dump` 
section.

I mainly want to avoid even more output code in the passes but also be able

to collect at least that information. That doesn't mean we couldn't add 
another

output stream that would always/only redirect into the sections.


~ Johannes


On 8/5/20 5:36 PM, Kazu Hirata via llvm-dev wrote:
> Introduction
> ===========>
> This RFC proposes a mechanism to dump arbitrary messages into object
> files during compilation and retrieve them from the final executable.
>
> Background
> =========>
> We often need to collect information from all object files of
> applications.  For example:
>
> - Mircea Trofin needs to collect information from the function
>    inlining pass so that he can train the machine learning model with
>    the information.
>
> - I sometimes need to dump messages from optimization passes to see
>    where and how they trigger.
>
> Now, this process becomes challenging when we build large applications
> with a build system that caches and distributes compilation jobs.  If
> we were to dump messages to stderr, we would have to be careful not to
> interleave messages from multiple object files.  If we were to modify
> a source file, we would have to flush the cache and rebuild the entire
> application to collect dump messages from all relevant object files.
>
> High Level Design
> ================>
> - LLVM: We provide machinery for individual passes to dump arbitrary
>    messages into a special ELF section in a compressed manner.
>
> - Linker: We simply concatenate the contents of the special ELF
>    section.  No change is needed.
>
> - llvm-readobj: We add an option to retrieve the contents of the
>    special ELF section.
>
> Detailed Design
> ==============>
> DumpAccumulator analysis pass
> -----------------------------
>
> We create a new analysis pass called DumpAccumulator.  We add the
> analysis pass right at the beginning of the pass pipeline.  The new
> analysis pass holds the dump messages throughout the pass pipeline.
>
> If you would like to dump messages from some pass, you would obtain
> the result of DumpAccumulator in the pass:
>
>    DumpAccumulator::Result *DAR =
MAMProxy.getCachedResult<DumpAccumulator>(M);
>
> Then dump messages:
>
>    if (DAR) {
>      DAR->Message += "Processing ";
>      DAR->Message += F.getName();
>      DAR->Message += "\n";
>    }
>
> AsmPrinter
> ----------
>
> We dump the messages from DumpAccumulator into a section called
> ".llvm_dump" in a compressed manner.  Specifically, the section
> contains:
>
> - LEB128 encoding of the original size in bytes
> - LEB128 encoding of the compressed size in bytes
> - the message compressed by zlib::compressed
>
> in that order.
>
> llvm-readobj
> ------------
>
> We read the .llvm_dump section.  We dump each chunk of compressed data
> one after another.
>
> Existing Implementation
> ======================> https://reviews.llvm.org/D84473
>
> Future Directions
> ================>
> The proposal above does not support the ThinLTO build flow.  To
> support that, I am thinking about putting the message as metadata in
> the IR at the prelink stage.
>
> Thoughts?
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On Wed, Aug 5, 2020 at 3:51 PM Eli Friedman <efriedma at quicinc.com>
wrote:
> I’m not a fan of keeping important data outside the IR in an analysis.  If
> we’re planning to emit it, it should be represented directly in the IR.  Is
> there some reason we can’t just stick the data in a global variable?
>
The analysis in the scenarios here is external to LLVM - ML training, for
example. It's really a way to do printf, but where the data could be large
(challenging in a distributed build env, where IO may be throttled), or
non-textual (for instance, capture IR right before a pass). An alternative
would be to produce a side-file, but then (again, distributed build), you
have to collect those files and concatenate them, and modify the build
system to be aware of all that.

>
> I’m not sure it’s helpful to have a generic mechanism for this; it’s not
> clear how this would work if multiple different features were trying to
> emit data into the llvm_dump section at the same time.
>
You could layer the approach: the one llvm_dump section has a pluggable
reader.

>
>
> -Eli
>
>
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of
*Kazu
> Hirata via llvm-dev
> *Sent:* Wednesday, August 5, 2020 3:36 PM
> *To:* llvm-dev at lists.llvm.org; Mircea Trofin <mtrofin at
google.com>; Wei Mi
> <wmi at google.com>; james.henderson at sony.com
> *Subject:* [EXT] [llvm-dev] [RFC] Introduce Dump Accumulator
>
>
>
> Introduction
>
> ===========>
>
>
> This RFC proposes a mechanism to dump arbitrary messages into object
>
> files during compilation and retrieve them from the final executable.
>
>
>
> Background
>
> =========>
>
>
> We often need to collect information from all object files of
>
> applications.  For example:
>
>
>
> - Mircea Trofin needs to collect information from the function
>
>   inlining pass so that he can train the machine learning model with
>
>   the information.
>
>
>
> - I sometimes need to dump messages from optimization passes to see
>
>   where and how they trigger.
>
>
>
> Now, this process becomes challenging when we build large applications
>
> with a build system that caches and distributes compilation jobs.  If
>
> we were to dump messages to stderr, we would have to be careful not to
>
> interleave messages from multiple object files.  If we were to modify
>
> a source file, we would have to flush the cache and rebuild the entire
>
> application to collect dump messages from all relevant object files.
>
>
>
> High Level Design
>
> ================>
>
>
> - LLVM: We provide machinery for individual passes to dump arbitrary
>
>   messages into a special ELF section in a compressed manner.
>
>
>
> - Linker: We simply concatenate the contents of the special ELF
>
>   section.  No change is needed.
>
>
>
> - llvm-readobj: We add an option to retrieve the contents of the
>
>   special ELF section.
>
>
>
> Detailed Design
>
> ==============>
>
>
> DumpAccumulator analysis pass
>
> -----------------------------
>
>
>
> We create a new analysis pass called DumpAccumulator.  We add the
>
> analysis pass right at the beginning of the pass pipeline.  The new
>
> analysis pass holds the dump messages throughout the pass pipeline.
>
>
>
> If you would like to dump messages from some pass, you would obtain
>
> the result of DumpAccumulator in the pass:
>
>
>
>   DumpAccumulator::Result *DAR =
MAMProxy.getCachedResult<DumpAccumulator>(M);
>
>
>
> Then dump messages:
>
>
>
>   if (DAR) {
>
>     DAR->Message += "Processing ";
>
>     DAR->Message += F.getName();
>
>     DAR->Message += "\n";
>
>   }
>
>
>
> AsmPrinter
>
> ----------
>
>
>
> We dump the messages from DumpAccumulator into a section called
>
> ".llvm_dump" in a compressed manner.  Specifically, the section
>
> contains:
>
>
>
> - LEB128 encoding of the original size in bytes
>
> - LEB128 encoding of the compressed size in bytes
>
> - the message compressed by zlib::compressed
>
>
>
> in that order.
>
>
>
> llvm-readobj
>
> ------------
>
>
>
> We read the .llvm_dump section.  We dump each chunk of compressed data
>
> one after another.
>
>
>
> Existing Implementation
>
> ======================>
>
>
> https://reviews.llvm.org/D84473
>
>
>
> Future Directions
>
> ================>
>
>
> The proposal above does not support the ThinLTO build flow.  To
>
> support that, I am thinking about putting the message as metadata in
>
> the IR at the prelink stage.
>
>
>
> Thoughts?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200805/b0e3a8dd/attachment.html>

I think that we should think about the relationship between this 
proposed mechanism and the existing mechanism that we have for emitting 
and capturing optimization remarks. In some sense, I feel like we 
already have a lot of this capability (e.g., llc has -remarks-section).

  -Hal

On 8/5/20 5:51 PM, Johannes Doerfert via llvm-dev wrote:
> I like the ability, not sure about the proposed implementation though.
>
> Did you consider a flag that redirects `llvm::outs()` and `llvm::errs()`
>
> into sections of the object file instead? So, you'd say:
>
>
> `clang ... -mllvm -debug-only=inline ... -mllvm -dump-section=.dump`
>
>
> and you'd get the regular debug output nicely ordered in the `.dump` 
> section.
>
> I mainly want to avoid even more output code in the passes but also be 
> able
>
> to collect at least that information. That doesn't mean we couldn't
> add another
>
> output stream that would always/only redirect into the sections.
>
>
> ~ Johannes
>
>
> On 8/5/20 5:36 PM, Kazu Hirata via llvm-dev wrote:
>> Introduction
>> ===========>>
>> This RFC proposes a mechanism to dump arbitrary messages into object
>> files during compilation and retrieve them from the final executable.
>>
>> Background
>> =========>>
>> We often need to collect information from all object files of
>> applications.  For example:
>>
>> - Mircea Trofin needs to collect information from the function
>>    inlining pass so that he can train the machine learning model with
>>    the information.
>>
>> - I sometimes need to dump messages from optimization passes to see
>>    where and how they trigger.
>>
>> Now, this process becomes challenging when we build large applications
>> with a build system that caches and distributes compilation jobs.  If
>> we were to dump messages to stderr, we would have to be careful not to
>> interleave messages from multiple object files.  If we were to modify
>> a source file, we would have to flush the cache and rebuild the entire
>> application to collect dump messages from all relevant object files.
>>
>> High Level Design
>> ================>>
>> - LLVM: We provide machinery for individual passes to dump arbitrary
>>    messages into a special ELF section in a compressed manner.
>>
>> - Linker: We simply concatenate the contents of the special ELF
>>    section.  No change is needed.
>>
>> - llvm-readobj: We add an option to retrieve the contents of the
>>    special ELF section.
>>
>> Detailed Design
>> ==============>>
>> DumpAccumulator analysis pass
>> -----------------------------
>>
>> We create a new analysis pass called DumpAccumulator.  We add the
>> analysis pass right at the beginning of the pass pipeline.  The new
>> analysis pass holds the dump messages throughout the pass pipeline.
>>
>> If you would like to dump messages from some pass, you would obtain
>> the result of DumpAccumulator in the pass:
>>
>>    DumpAccumulator::Result *DAR = 
>> MAMProxy.getCachedResult<DumpAccumulator>(M);
>>
>> Then dump messages:
>>
>>    if (DAR) {
>>      DAR->Message += "Processing ";
>>      DAR->Message += F.getName();
>>      DAR->Message += "\n";
>>    }
>>
>> AsmPrinter
>> ----------
>>
>> We dump the messages from DumpAccumulator into a section called
>> ".llvm_dump" in a compressed manner.  Specifically, the
section
>> contains:
>>
>> - LEB128 encoding of the original size in bytes
>> - LEB128 encoding of the compressed size in bytes
>> - the message compressed by zlib::compressed
>>
>> in that order.
>>
>> llvm-readobj
>> ------------
>>
>> We read the .llvm_dump section.  We dump each chunk of compressed data
>> one after another.
>>
>> Existing Implementation
>> ======================>> https://reviews.llvm.org/D84473
>>
>> Future Directions
>> ================>>
>> The proposal above does not support the ThinLTO build flow.  To
>> support that, I am thinking about putting the message as metadata in
>> the IR at the prelink stage.
>>
>> Thoughts?
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

One area the proposal doesn't cover is the ELF file format properties.
What section type will this be? What flags will it have? Is the output
intended to be part of the executable loadable image or not? Does it even
need to be in the final executable output, or is it just being in the
object sufficient? The same or similar questions arise if you are
considering this for other output formats (e.g. COFF, Mach-O etc) for each
of them too.

On Wed, 5 Aug 2020 at 23:36, Kazu Hirata via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Introduction
> ===========>
> This RFC proposes a mechanism to dump arbitrary messages into object
> files during compilation and retrieve them from the final executable.
>
> Background
> =========>
> We often need to collect information from all object files of
> applications.  For example:
>
> - Mircea Trofin needs to collect information from the function
>   inlining pass so that he can train the machine learning model with
>   the information.
>
> - I sometimes need to dump messages from optimization passes to see
>   where and how they trigger.
>
> Now, this process becomes challenging when we build large applications
> with a build system that caches and distributes compilation jobs.  If
> we were to dump messages to stderr, we would have to be careful not to
> interleave messages from multiple object files.  If we were to modify
> a source file, we would have to flush the cache and rebuild the entire
> application to collect dump messages from all relevant object files.
>
> High Level Design
> ================>
> - LLVM: We provide machinery for individual passes to dump arbitrary
>   messages into a special ELF section in a compressed manner.
>
> - Linker: We simply concatenate the contents of the special ELF
>   section.  No change is needed.
>
> - llvm-readobj: We add an option to retrieve the contents of the
>   special ELF section.
>
> Detailed Design
> ==============>
> DumpAccumulator analysis pass
> -----------------------------
>
> We create a new analysis pass called DumpAccumulator.  We add the
> analysis pass right at the beginning of the pass pipeline.  The new
> analysis pass holds the dump messages throughout the pass pipeline.
>
> If you would like to dump messages from some pass, you would obtain
> the result of DumpAccumulator in the pass:
>
>   DumpAccumulator::Result *DAR =
MAMProxy.getCachedResult<DumpAccumulator>(M);
>
> Then dump messages:
>
>   if (DAR) {
>     DAR->Message += "Processing ";
>     DAR->Message += F.getName();
>     DAR->Message += "\n";
>   }
>
> AsmPrinter
> ----------
>
> We dump the messages from DumpAccumulator into a section called
> ".llvm_dump" in a compressed manner.  Specifically, the section
> contains:
>
> - LEB128 encoding of the original size in bytes
> - LEB128 encoding of the compressed size in bytes
> - the message compressed by zlib::compressed
>
> in that order.
>
> llvm-readobj
> ------------
>
> We read the .llvm_dump section.  We dump each chunk of compressed data
> one after another.
>
> Existing Implementation
> ======================> https://reviews.llvm.org/D84473
>
> Future Directions
> ================>
> The proposal above does not support the ThinLTO build flow.  To
> support that, I am thinking about putting the message as metadata in
> the IR at the prelink stage.
>
> Thoughts?
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200806/e0afd508/attachment.html>