[llvm-dev] Commit module to Git after each Pass

Alexandre Isoard via llvm-dev
2018-03-15T15:09:40Z

Does git-commit-after-all print correctly after all the passes? Maybe I
messed it up and it skip some passes, therefore having less to do?

Either that, or piping has a higher cost than writing to file. Looks like
it surprisingly spends much less time in system more when going through
file. Maybe that's because the file is consistently around the same size
and is mmapped into memory continuously, while piping require regular (more
than once per module) context switches between the two processes?

Honestly, I would say something is wrong (aka. first paragraph). I didn't
build that with efficiency in mind in any way...

On Thu, Mar 15, 2018, 07:47 Fedor Sergeev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hmm...
>
> I tried Alexandre's fix from D44244 and surprisingly it appears that
> just using -print-module-scope w/o
> any additional git actions is waaaay slower on my testcase than
> -git-commit-module-all.
>
> Hell, even a plan -print-after-all is slower:
>
>   ] time R/bin/opt -O3 some-ir.ll -disable-output -git-commit-after-all
> 2>/dev/null
> real    0m8.041s
> user    0m7.133s
> sys     0m0.936s
> ] time R/bin/opt -O3 some-ir.ll -disable-output -print-after-all
> 2>/dev/null
>
> real    0m13.575s
> user    0m6.179s
> sys     0m7.394s
>
> I cant really explain that...
>
> regards,
>    Fedor.
>
> On 03/15/2018 04:30 PM, Fedor Sergeev via llvm-dev wrote:
> >
> >
> > On 03/15/2018 01:32 PM, Fedor Sergeev via llvm-dev wrote:
> >> For this to be really usable in this setup we need additionally
to:
> >>   - extend -print-module-scope to cover basic block passes
> >>   - introduce a clear way to separate module IRs as those are
being
> >> printed by -print-after-all
> >>
> >> But yes, it should work, and a wrapper that pipes to git
fast-import
> >> seems to be the best way to handle it.
> > A simple 20-lines perl script does the trick pretty easily:
> > https://pastebin.com/4J0b5Tr8
> >
> > (this assumes my local modification to introduce the *** END OF ** IR
> > DUMP marked at the end of -print-module-scope's IR module dump)
> >
> > ] git init
> > ] RA/bin/opt -O3 some-ir.ll -disable-output -print-after-all
> > -print-module-scope 2>&1 | filter-LLVM-ir-print.pl | git
fast-import
> > --done --date-format=now
> > ....
> >
> > Majority of time is spent to actually print the IR (~2m for my
testcase).
> > Fast-import takes just a second.
> >
> > regards,
> >   Fedor.
> >
> >>
> >> regards,
> >>   Fedor.
> >>
> >> On 03/15/2018 12:31 AM, Daniel Neilson via llvm-dev wrote:
> >>> The print-module-after-all type of option exists in upstream:
> >>>   -print-module-scope                                     -
When
> >>> printing IR for print-[before|after]{-all} always print a
module IR
> >>>
> >>> commit 7d160f714357f6784ead669ce516e94991c12e5a
> >>> Author: Fedor Sergeev <fedor.sergeev at azul.com
> >>> <mailto:fedor.sergeev at azul.com>>
> >>> Date:   Fri Dec 1 17:42:46 2017 +0000
> >>>
> >>>     IR printing improvement for function passes - introducing
> >>> -print-module-scope
> >>>
> >>>
> >>>     Summary:
> >>>     When debugging function passes it happens to be rather
useful to
> >>> dump
> >>>     the whole module before the transformation and then use
this dump
> >>>     to analyze this single transformation by running it
separately
> >>>     on that particular module state.
> >>>
> >>>
> >>>     Introducing
> >>>         -print-module-scope
> >>>     debugging option that forces all the function-level IR
dumps
> >>>     to become whole-module dumps.
> >>>
> >>>
> >>>     This option builds on top of normal dumping controls like
> >>>        -print-before/after
> >>>        -filter-print-funcs
> >>>
> >>>
> >>>     The plan is to eventually extend this option to cover
other
> >>> local passes
> >>>     (at least loop passes) but that should go as a separate
change.
> >>>
> >>>
> >>> Loop passes here:
> >>> commit 5608259c999fb77c5d6093895696f4daebe6b8cd
> >>> Author: Fedor Sergeev <fedor.sergeev at azul.com
> >>> <mailto:fedor.sergeev at azul.com>>
> >>> Date:   Fri Dec 1 18:33:58 2017 +0000
> >>>
> >>>     IR printing improvement for loop passes - handle
> >>> -print-module-scope
> >>>
> >>>
> >>> -Daniel
> >>>
> >>>
> >>>> On Mar 14, 2018, at 3:51 PM, Philip Reames via llvm-dev
> >>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
> >>>>
> >>>> This is interesting, and might be useful.  I don't
know that this
> >>>> is broadly useful enough for upstream inclusion, but if
you could
> >>>> post this to github somewhere, I might play with it.
> >>>>
> >>>> There might also be room to factor out common
functionality. We've
> >>>> also run into the need to print whole-module instead of
containing
> >>>> construct (i.e. this loop).  If we added upstream support
for
> >>>> something along the lines of -print-module-after-all,
building the
> >>>> git history could easily be done as a post processing
step.
> >>>>
> >>>> Philip
> >>>>
> >>>>
> >>>> On 03/06/2018 10:43 AM, Alexandre Isoard via llvm-dev
wrote:
> >>>>> Hello,
> >>>>>
> >>>>> I had a stupid idea recently that turned out not so
stupid after
> >>>>> all. I wanted to be able to "see" an entire
pass pipeline in
> >>>>> action to find unnecessary transformations and/or
missed
> >>>>> opportunities and generally improve the debug-ability
of LLVM.
> >>>>>
> >>>>> So as the title suggest, I implemented an equivalent
of
> >>>>> "-print-after-all" but instead of printing
into stdout I dump into
> >>>>> a file that get commit into a temporary git. There are
some quirks
> >>>>> with it but it's working and is actually awesome.
For example, at
> >>>>> first sight, I see multiple time lcssa and instcombine
cancelling
> >>>>> each other's work.
> >>>>>
> >>>>> Of course, that has a big impact on compile time when
enabled, but
> >>>>> that's still practical (git being quite good at
its job) when
> >>>>> debugging.
> >>>>>
> >>>>> There are improvement I can make, but would you guys
be interested
> >>>>> in such feature?
> >>>>>
> >>>>> --
> >>>>> *Alexandre Isoard*
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> LLVM Developers mailing list
> >>>>> llvm-dev at lists.llvm.org
> >>>>>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>>
> >>>> _______________________________________________
> >>>> LLVM Developers mailing list
> >>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
> >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180315/b1e3af1a/attachment.html>