similar to: Commit module to Git after each Pass

This is interesting, and might be useful.  I don't know that this is broadly useful enough for upstream inclusion, but if you could post this to github somewhere, I might play with it. There might also be room to factor out common functionality. We've also run into the need to print whole-module instead of containing construct (i.e. this loop).  If we added upstream support for

The print-module-after-all type of option exists in upstream: -print-module-scope - When printing IR for print-[before|after]{-all} always print a module IR commit 7d160f714357f6784ead669ce516e94991c12e5a Author: Fedor Sergeev <fedor.sergeev at azul.com<mailto:fedor.sergeev at azul.com>> Date: Fri Dec 1 17:42:46 2017 +0000 IR

On 03/15/2018 01:32 PM, Fedor Sergeev via llvm-dev wrote: > For this to be really usable in this setup we need additionally to: >   - extend -print-module-scope to cover basic block passes >   - introduce a clear way to separate module IRs as those are being > printed by -print-after-all > > But yes, it should work, and a wrapper that pipes to git fast-import > seems to be

For this to be really usable in this setup we need additionally to:   - extend -print-module-scope to cover basic block passes   - introduce a clear way to separate module IRs as those are being printed by -print-after-all But yes, it should work, and a wrapper that pipes to git fast-import seems to be the best way to handle it. regards,   Fedor. On 03/15/2018 12:31 AM, Daniel Neilson via

Hmm... I tried Alexandre's fix from D44244 and surprisingly it appears that just using -print-module-scope w/o any additional git actions is waaaay slower on my testcase than -git-commit-module-all. Hell, even a plan -print-after-all is slower:  ] time R/bin/opt -O3 some-ir.ll -disable-output -git-commit-after-all 2>/dev/null real    0m8.041s user    0m7.133s sys     0m0.936s ] time

Does git-commit-after-all print correctly after all the passes? Maybe I messed it up and it skip some passes, therefore having less to do? Either that, or piping has a higher cost than writing to file. Looks like it surprisingly spends much less time in system more when going through file. Maybe that's because the file is consistently around the same size and is mmapped into memory

Does https://reviews.llvm.org/D44132 <https://reviews.llvm.org/D44132> help at all? > On 15 Mar 2018, at 09:16, Philip Reames via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > The most likely answer is that the printer used by print-after-all is slow. I know there were some changes made around passing in some form of state cache (metadata related?) and that running

git-commit-after-all solution has one serious issue - it has a hardcoded git handling which makes it look problematic from many angles (picking a proper git, selecting exact way of storing information, creating repository, replacing the file etc etc). Just dumping information in a way that allows easy subsequent machine processing seems to be a more flexible, less cluttered and overall clean

Huh. Great! 😁 I don't believe my poor excuse from earlier (else we should map all pipes into files!), but I'm curious why we spend less time in system mode when going through file than pipe. Maybe /dev/null is not as efficient as we might think? I can't believe I'm saying that... On Thu, Mar 15, 2018, 08:25 Fedor Sergeev <fedor.sergeev at azul.com> wrote: > Well, git by

If this is faster than -print-after-all we may actually consider pushing that in the code base then? (after diligent code review of course) Note that it uses the same printing method as -print-after-all: - create a pass of the same pass kind as the pass we just ran - use Module::print(raw_ostream) to print (except -print-after-all only print the concerned part and into stdout) If there is

Hello, Just an update on that. I am personally using -git-commit-after-all *as-is* extremely frequently (combined with "git filter-branch" and "opt -S -instnamer" it is extremely useful). I unfortunately won't have time to write a better implementation of that, and I agree "git fast-import" seems the way to go. If anybody is motivated enough to do so, feel free.

On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote: > git-commit-after-all solution has one serious issue - it has a hardcoded git handling which > makes it look problematic from many angles (picking a proper git, > selecting exact way of storing information, creating repository, replacing the file etc etc). > > Just dumping information in a way that allows easy

On 03/15/2018 06:09 PM, Alexandre Isoard wrote: > Does git-commit-after-all print correctly after all the passes? Maybe > I messed it up and it skip some passes, therefore having less to do? I did verify that total amount of lines committed to git is reasonably high: ] git rev-list master | while read cmt; do git show $cmt:some-ir.ll; done | wc -l 1587532 corresponding number for

Oh, well... as usually the answer appears to be pretty obvious. 99% of the time is spent inside the plain write. -print-after-all prints into llvm::errs(), which is an *unbuffered* raw_fd_stream. And -git-commit-after-all opens a *buffered* raw_fd_stream. As soon as I hacked -print-after-all to use a buffered stream to stderr performance went up to the normal expected values: ] time bin/opt

Do you really need to write the entire module to a single file? (Hence my earlier hint...) Why not write out a separate file for each def, so you don't need to dump functions that haven't changed? On Wed, Mar 21, 2018 at 8:38 PM, Fedor Sergeev via llvm-dev < llvm-dev at lists.llvm.org> wrote: > On 03/16/2018 01:21 AM, Fedor Sergeev via llvm-dev wrote: > >

The most likely answer is that the printer used by print-after-all is slow.  I know there were some changes made around passing in some form of state cache (metadata related?) and that running printers without doing so work, but are dog slow.  I suspect the print-after-all support was never updated.  Look at what we do for the normal IR emission "-S" and see if print-after-all is

Today it is entirely in llvm. It is even more costly than -print-after-all as it: - print to file - print the entire module (after each basic block, for basic block passes, it will still print the entire module where only one basic block changed) - call git 2 times (add then commit) and wait for them to finish (I even save all the empty commits) The reason I print the entire module is so

On 6 March 2018 at 18:43, Alexandre Isoard via llvm-dev <llvm-dev at lists.llvm.org> wrote: > Hello, > > I had a stupid idea recently that turned out not so stupid after all. I > wanted to be able to "see" an entire pass pipeline in action to find > unnecessary transformations and/or missed opportunities and generally > improve the debug-ability of LLVM. > >

Maybe. I can't compute the diff/patch for git though, it will have to do it itself. Would that still work? I am not sure I will have time to work on improving that. For those who want to try it out I put it there: https://reviews.llvm.org/D44244 And yes, this uses all possible forms of ugly. :-) On Wed, Mar 7, 2018, 18:14 Jeremy Lakeman <Jeremy.Lakeman at gmail.com> wrote: > Could

Could you format the output so that it is compatible with git fast-import? https://www.git-scm.com/docs/git-fast-import On Thu, Mar 8, 2018 at 2:34 AM, Alexandre Isoard via llvm-dev < llvm-dev at lists.llvm.org> wrote: > Today it is entirely in llvm. It is even more costly than -print-after-all > as it: > - print to file > - print the entire module (after each basic block,