search for: lto'ing

On Fri, Feb 05, 2016 at 04:58:45PM -0800, Mehdi Amini wrote: > > > On Feb 5, 2016, at 3:17 PM, Peter Collingbourne via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, and > > we've found that one of the top consumers of memory is the DWARF emitter in > > lib/CodeGen/AsmPrinter/Dwarf*. > > I'm staring at the profile attached to the post #15 on the link you posted, can you confirm that the Dwarf emitter accounts for 6.7...

...that in the backend, we emit a function at a time, from DAG formation to object emission. So no two MachineFunctions ever exist at any one time. Changing this necessarily means increasing memory usage. I've prototyped this change and have measured peak memory usage in the worst case scenario - LTO'ing llc and clang. Without further ado: llvm-lto llc: before: 1.44GB maximum resident set size after: 1.68GB (+17%) llvm-lto clang: before: 2.48GB maximum resident set size after: 3.42GB (+33%) The increases are very large. This is worst-case (non-...

Hi all, We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, and we've found that one of the top consumers of memory is the DWARF emitter in lib/CodeGen/AsmPrinter/Dwarf*. I've been reading the DWARF emitter code and I have a few ideas in mind for how to reduce its memory consumption. One idea I've had is to restructure the emit...

...M -0800, Mehdi Amini wrote: > >> > >>> On Feb 5, 2016, at 3:17 PM, Peter Collingbourne via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >>> > >>> Hi all, > >>> > >>> We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, > and > >>> we've found that one of the top consumers of memory is the DWARF > emitter in > >>> lib/CodeGen/AsmPrinter/Dwarf*. > >> > >> I'm staring at the profile attached to the post #15 on the link you > posted, can you...

...>> >> >> >>> On Feb 5, 2016, at 3:17 PM, Peter Collingbourne via llvm-dev < >> llvm-dev at lists.llvm.org> wrote: >> >>> >> >>> Hi all, >> >>> >> >>> We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, >> and >> >>> we've found that one of the top consumers of memory is the DWARF >> emitter in >> >>> lib/CodeGen/AsmPrinter/Dwarf*. >> >> >> >> I'm staring at the profile attached to the post #15 on the link...

Hi all, I like all the ideas so far. Here are my thoughts: I think that fundamentally users of LLVM should be able to opt-in to more aggressive or intensive computation at compile time if they wish. Users' needs differ, and while a 33% increase in clang LTO is absolutely out of the question for some people, for those developing microcontrollers or HPC applications that may well be irrelevant. Either the volume of code expected is significantly smaller or they're happy to trade off compile time for expensive server time. That does not mean that we...

>> In the case of user-defined allocation functions, the definitions for those functions are available >Are they? probably not unless you're in an LTO build. Yes, I'm assuming an LTO build. >Printf() is a very nasty one because it can actually affect a lot of state. The %n modifier can cause an argument to be written to. hence it would have HasInaccessibleState and ArgMemOnly set, but not ReadNone or ReadOnly set. >Yes. Definitions be...

...tions, much like we already do, but restrict it to external functions. - Recognize internal functions *with a builtin attribute* much like we do external library functions. - Teach internalize to add the builtin attribute as it changes linkage. One example of what I *really* want from this even in LTO which motivates the change to internalize: things like 'readonly' where some spec lets us optimize callers with this even if the implementation actually writes to memory. Consider building with -fno-math-errno and LTOing a libc that does actually set errno in its implementation. We will al...

Due to the LTO modifications to the build system it is not possible to do old non LTO builds that involve some old PIC behaviour, this means Cygwin and possible other platforms old behaviours is broken, and they are not LTO'ed yet either. Would it be possible to have both the old PIC behaviour and the new LT...

...gt; if that's the case we'd be able to be as aggressive as I described is the > case for fission > > On Fri, Feb 5, 2016 at 3:17 PM, Peter Collingbourne <peter at pcc.me.uk> wrote: > > > Hi all, > > > > We have profiled [1] the memory usage in LLVM when LTO'ing Chromium, and > > we've found that one of the top consumers of memory is the DWARF emitter in > > lib/CodeGen/AsmPrinter/Dwarf*. I've been reading the DWARF emitter code and > > I have a few ideas in mind for how to reduce its memory consumption. One > > idea...

...t; Typically consist of Loop-Nest-Opt, Scalar Opt, Code-Gen etc etc. > While they > are intra-procedural analyses/optimizers, they may directly benefit > from > the info collected in the IPO stages and pass down the road. > > LLVM collectively call S2 and S3 as "LTO CodeGen", which is very > confusing. > > 2. Why parallelize post-IPO stage > ============================== > > R1) To improve the scalarbility > It is quite obvious that we are not able to put everything about a > monster > program in memory at once. >...

...s with the measurements in that blog > post. I built LLD with the RelWithDebInfo configuration which we later > found out uses /Ob1 instead of /Ob2. That was worth some cycles. Then > there were some more optimizations that went in after that. And to get > down to 28s I also used an LTO'ed build of lld. > > If you're building LLD at ToT you should have everything needed to > reproduce those numbers, but it will vary depending on the speed of your > CPU obviously. > > On Tue, Jan 30, 2018 at 9:21 AM Leonardo Santagada <santagada at gmail.com> > wr...

...re were some problems with the measurements in that blog post. I built LLD with the RelWithDebInfo configuration which we later found out uses /Ob1 instead of /Ob2. That was worth some cycles. Then there were some more optimizations that went in after that. And to get down to 28s I also used an LTO'ed build of lld. If you're building LLD at ToT you should have everything needed to reproduce those numbers, but it will vary depending on the speed of your CPU obviously. On Tue, Jan 30, 2018 at 9:21 AM Leonardo Santagada <santagada at gmail.com> wrote: > Today I played around...

...rements in that blog >> post. I built LLD with the RelWithDebInfo configuration which we later >> found out uses /Ob1 instead of /Ob2. That was worth some cycles. Then >> there were some more optimizations that went in after that. And to get >> down to 28s I also used an LTO'ed build of lld. >> >> If you're building LLD at ToT you should have everything needed to >> reproduce those numbers, but it will vary depending on the speed of your >> CPU obviously. >> >> On Tue, Jan 30, 2018 at 9:21 AM Leonardo Santagada <santagada...

...blog >>> post. I built LLD with the RelWithDebInfo configuration which we later >>> found out uses /Ob1 instead of /Ob2. That was worth some cycles. Then >>> there were some more optimizations that went in after that. And to get down >>> to 28s I also used an LTO'ed build of lld. >>> >>> If you're building LLD at ToT you should have everything needed to >>> reproduce those numbers, but it will vary depending on the speed of your CPU >>> obviously. >>> >>> On Tue, Jan 30, 2018 at 9:21 AM Leonardo...

...st. I built LLD with the RelWithDebInfo configuration which we later > >>> found out uses /Ob1 instead of /Ob2. That was worth some cycles. Then > >>> there were some more optimizations that went in after that. And to > get down > >>> to 28s I also used an LTO'ed build of lld. > >>> > >>> If you're building LLD at ToT you should have everything needed to > >>> reproduce those numbers, but it will vary depending on the speed of > your CPU > >>> obviously. > >>> > >>> On Tue...

...>>> later >> >>> found out uses /Ob1 instead of /Ob2. That was worth some cycles. >> >>> Then >> >>> there were some more optimizations that went in after that. And to >> >>> get down >> >>> to 28s I also used an LTO'ed build of lld. >> >>> >> >>> If you're building LLD at ToT you should have everything needed to >> >>> reproduce those numbers, but it will vary depending on the speed of >> >>> your CPU >> >>> obviously. >>...

...>>>> >>>>>>> >>>>>>> On Sun, Jan 28, 2018 at 11:22 PM, Zachary Turner <zturner at google.com >>>>>>> > wrote: >>>>>>> >>>>>>>> I don’t have pgo numbers. When I build using -flto=thin the link >>>>>>>> time is significantly faster than msvc /ltcg and runtime is slightly >>>>>>>> faster, but I haven’t tested on a large variety of different workloads, so >>>>>>>> YMMV. Link time will definitely be faster...

...#3 is a real issue. The only case I have found that actually hits #3 at all hits #3b when building FORTIFY code with the new pass manager because after inlining we do a lot of (really nice) optimizations on library calls to remove unnecessary FORTIFY checks. But this is in *theory* a problem when LTO-ing with libc. More likely it could be a problem when LTO-ing with a vector math library. So what do we do? My initial idea: find all *defined* library functions in the module, and every time we create a ref edge to one of them, synthesize a ref edge to all of them. This should completely solve #...

...;>>>>>>> >>>>>>>> On Sun, Jan 28, 2018 at 11:22 PM, Zachary Turner < >>>>>>>> zturner at google.com> wrote: >>>>>>>> >>>>>>>>> I don’t have pgo numbers. When I build using -flto=thin the link >>>>>>>>> time is significantly faster than msvc /ltcg and runtime is slightly >>>>>>>>> faster, but I haven’t tested on a large variety of different workloads, so >>>>>>>>> YMMV. Link time will definitel...