llvm dev - Jan 2016 - High memory use and LVI/Correlated Value Propagation

Hi all,
with the current trunk I have two major cases where clang needs more
than 2GB memory for compiling programs with -O2. One is related to GVN
and MemoryDependenceAnalysis and has a pending patch. The other is
related to the Correlated Value Propagation and Lazy Value Information
cache. Attached is a heap profile for one of the relevant test cases.
Looking at the sources, I don't see any form of CFG limiters in LVI or
CVP, so it doesn't seem surprising that large memory use can be easily
triggered. This normally also corresponds to very slow compilation, so
it addressing it fixes two issues at the same time. A test case can be
found in https://llvm.org/bugs/show_bug.cgi?id=10584.

The question for me is: where should such complexity limits be placed --
in the CVP pass or inside LVI? Is anyone specifically interested in
hunting down the problem?

Joerg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: expr.pdf
Type: application/pdf
Size: 12311 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160113/f82bd38f/attachment.pdf>

I don't think that arbitrary limiting the complexity of the search is 
the right approach.  There are numerous ways the LVI infrastructure 
could be made more memory efficient.  Fixing the existing code to be 
memory efficient is the right approach.  Only once there's no more low 
hanging fruit should we even consider clamping the search.

In general, introducing clamps should be an absolute last resort. We 
really don't want an optimizer than if fragile with respect to code size 
and shape.

Philip

On 01/13/2016 02:27 PM, Joerg Sonnenberger via llvm-dev
wrote:
> Hi all,
> with the current trunk I have two major cases where clang needs more
> than 2GB memory for compiling programs with -O2. One is related to GVN
> and MemoryDependenceAnalysis and has a pending patch. The other is
> related to the Correlated Value Propagation and Lazy Value Information
> cache. Attached is a heap profile for one of the relevant test cases.
> Looking at the sources, I don't see any form of CFG limiters in LVI or
> CVP, so it doesn't seem surprising that large memory use can be easily
> triggered. This normally also corresponds to very slow compilation, so
> it addressing it fixes two issues at the same time. A test case can be
> found in https://llvm.org/bugs/show_bug.cgi?id=10584.
>
> The question for me is: where should such complexity limits be placed --
> in the CVP pass or inside LVI? Is anyone specifically interested in
> hunting down the problem?
>
> Joerg
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160113/db215a7a/attachment.html>

On Wed, Jan 13, 2016 at 03:38:24PM -0800, Philip Reames
wrote:
> I don't think that arbitrary limiting the complexity of the search is
the
> right approach.  There are numerous ways the LVI infrastructure could be
> made more memory efficient.  Fixing the existing code to be memory
efficient
> is the right approach.  Only once there's no more low hanging fruit
should
> we even consider clamping the search.
Memory efficiency is only half of the problem. I.e. groonga's expr.c
needs 4m to build on my laptop, a 2.7GHz i7. That doesn't sound
reasonable for a -O2. Unlike the GVN issue, the cases I have run into do
finish after a(n unreasonable) while, so at least it is not trivially
superlinear.

Joerg

Hi Joerg,
> with the current trunk I have two major cases where clang needs more
> than 2GB memory for compiling programs with -O2. One is related to GVN
> and MemoryDependenceAnalysis and has a pending patch. The other is
> related to the Correlated Value Propagation and Lazy Value Information
> cache.
For the LVI part, how recent are you measurements? I wonder whether
Akira's r255320 helps.

r255320 => [LazyValueInfo] Stop inserting overdefined values into
ValueCache to reduce memory usage.

-- 
Bruno Cardoso Lopes
http://www.brunocardoso.cc

Did it take an hour to compile the file attached to PR10584 (hugemem.ii)?

It took about 4 minutes and about 450MB of memory on my machine to compile
the file with "clang++ -O2".

On Thu, Jan 14, 2016 at 9:56 AM, Bruno Cardoso Lopes via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Joerg,
>
> > with the current trunk I have two major cases where clang needs more
> > than 2GB memory for compiling programs with -O2. One is related to GVN
> > and MemoryDependenceAnalysis and has a pending patch. The other is
> > related to the Correlated Value Propagation and Lazy Value Information
> > cache.
>
> For the LVI part, how recent are you measurements? I wonder whether
> Akira's r255320 helps.
>
> r255320 => [LazyValueInfo] Stop inserting overdefined values into
> ValueCache to reduce memory usage.
>
> --
> Bruno Cardoso Lopes
> http://www.brunocardoso.cc
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160114/14e42658/attachment.html>

On Thu, Jan 14, 2016 at 09:56:38AM -0800, Bruno Cardoso Lopes
wrote:
> Hi Joerg,
> 
> > with the current trunk I have two major cases where clang needs more
> > than 2GB memory for compiling programs with -O2. One is related to GVN
> > and MemoryDependenceAnalysis and has a pending patch. The other is
> > related to the Correlated Value Propagation and Lazy Value Information
> > cache.
> 
> For the LVI part, how recent are you measurements? I wonder whether
> Akira's r255320 helps.
> 
> r255320 => [LazyValueInfo] Stop inserting overdefined values into
> ValueCache to reduce memory usage.
I've rebuilt all my test cases since the weekend. It's been getting
better and I have now down to Groonga, G'MIC, FET and Scribus as
triggering, which is much better.

Joerg