llvm dev - Aug 2020 - [RFC] Introduce Dump Accumulator

On Thu, Aug 6, 2020 at 1:47 AM Renato Golin <rengolin at gmail.com> wrote:
> On Thu, 6 Aug 2020 at 00:16, Mircea Trofin via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > The analysis in the scenarios here is external to LLVM - ML training,
> for example. It's really a way to do printf, but where the data could
be
> large (challenging in a distributed build env, where IO may be throttled),
> or non-textual (for instance, capture IR right before a pass). An
> alternative would be to produce a side-file, but then (again, distributed
> build), you have to collect those files and concatenate them, and modify
> the build system to be aware of all that.
>
> I don't understand why we'd add a functionality to LLVM that is
> external to LLVM and not want to change the build system accordingly.
>
> All of the needs you describe on your email can already be done using
> optimization remarks and dumped into a file.
>
> The main difference for you, in the distributed build, is that you
> rely on the linker to do the concatenation, and for that you change
> the compiler and the ELF files you produce.
>
At a high level, you are right, the 2 alternatives are similar, but the
devil is in the details. The build system (basel-based) is hermetic, and
needs to be aware of all such extra files, and have a separate rule to copy
and concatenate them. This solution turned out to be much cleaner.
>From Hal's earlier message, it seems we already have something along the
lines of what we need in the "-remarks-section"
(llvm/lib/Remarks/RemarkStreamer.cpp or llvm/docs/Remarks.rst), so I think
we need to investigate what, if anything, needs to be added to fit our
scenarios.

>
> Seems to me like a roundabout way of doing concatenation of debug
> messages that could easily be done by a simple script and added to
> your build system.
>
> In the past, when I wanted something similar, I wrote a small script
> that would be called as the compiler (CMAKE_C_COMPILER=myscript) and
> it would do the fiddling with return values and the outputs onto local
> fast storage. For a distributed build you'd need another script to
> copy them into your dispatcher (or something), and that's it.
>
> What am I missing?
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200806/6fef12a6/attachment-0001.html>

On Thu, 6 Aug 2020 at 16:49, Mircea Trofin <mtrofin at google.com>
wrote:
> At a high level, you are right, the 2 alternatives are similar, but the
devil is in the details. The build system (basel-based) is hermetic, and needs
to be aware of all such extra files, and have a separate rule to copy and
concatenate them. This solution turned out to be much cleaner.
Cleaner to your build system is not necessarily cleaner to LLVM and
all its users (downstream and upstream) and sub-projects.

What I'm trying to avoid is a custom-built solution injecting random
sections in the ELF binary to fix a problem that a specific build
system has for a specific project. That doesn't scale.

Like Hal, I'm trying to get you to use existing solutions in LLVM to
suit your needs, as the cost of keeping dangling features (ones used
by a single project) in a code as large as LLVM usually doesn't pay
off.

Even if that makes your build system slightly worse, the benefit of
not adding bespoke features, to all LLVM users (including you), is
still very much positive.

--renato

No disagreement there. We illustrate with scenarios we have only because we
can speak confidently about them, and the hope was they may resonate with
others' (we didn't expected them to be sufficient motivation, had they
been
ours alone). If it turned out that no one had similar scenarios, or the
design was too narrow, we would have sought an outside-llvm alternative,
for all the reasons you mention. In this case, RemarksEmitter seems to
suggest related problems were addressed already.

On Thu, Aug 6, 2020 at 9:17 AM Renato Golin <rengolin at gmail.com> wrote:
> On Thu, 6 Aug 2020 at 16:49, Mircea Trofin <mtrofin at google.com>
wrote:
> > At a high level, you are right, the 2 alternatives are similar, but
the
> devil is in the details. The build system (basel-based) is hermetic, and
> needs to be aware of all such extra files, and have a separate rule to copy
> and concatenate them. This solution turned out to be much cleaner.
>
> Cleaner to your build system is not necessarily cleaner to LLVM and
> all its users (downstream and upstream) and sub-projects.
>
> What I'm trying to avoid is a custom-built solution injecting random
> sections in the ELF binary to fix a problem that a specific build
> system has for a specific project. That doesn't scale.
>
> Like Hal, I'm trying to get you to use existing solutions in LLVM to
> suit your needs, as the cost of keeping dangling features (ones used
> by a single project) in a code as large as LLVM usually doesn't pay
> off.
>
> Even if that makes your build system slightly worse, the benefit of
> not adding bespoke features, to all LLVM users (including you), is
> still very much positive.
>
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200806/c7c3405f/attachment.html>