llvm dev - Jul 2010 - [LLVMdev] Function-at-a-time Processing

We process very large programs and it is not unusual for the IR for some
compilation unit to exceed system memory.  With some hacking in LLVM 2.5
I was able to coax LLVM to generate asm for each functioin as it was
processed and then completely forget about it (i.e. delete it) and move
on to the next function.

This required a bit of hackery.  I had to create two pass managers, one
for the module and one for each function.  I called doInitialization on
the module pass manager to convince the AsmPrinter to dump out global
information needed at the top of the asm file.

Then I would run each function though the function pass manager as they
were encountered.  I could construct, run and destroy the function pass
manager for each function.

Finally I would run the module pass manager and doFinalization on the
module pass manager to dump out the stuff needed at the end of the asm
file.

This was not at all pretty but I could at least get it to work.

I'm in the last stages of upgrading to LLVM 2.7 and I'm finding that the
new MCStreamer system is making this sort of model much more difficult.
MCContexts are somehow shared between the separate module and function
pass managers in ways I don't understand.  I think that somehow the
LLVMTargetMachine is shared between the pass managers.

I think this sort of sharing is actually correct since we want the
MCContext for the function pass manager to be the same as the one for
the module pass manager.  That way globals, etc. have consistent
representations in both.  Unfortunately, when the first function pass
manager gets deleted, it destroys the MCContext alonmg with it.
Actually, the AsmPrinter destroys the MCContext through a reference (!)
which makes things all the more confusing.  When the module pass manager
comes along and tries of access an MCSection, everything blows up.

I had to re-add addAssemblyEmitter to LLVMTargetMachine to try to make
this work.  It essentially takes the AsmPrinter creation bits from
addPassesToEmitFile and packages them into a separate function.

[As a side note, I don't understand how the two AsmPrinters (one for the
module pass manager, one for the function pass manager) end up pointing
to the same MCContext as LLVMTargetMachine::addPassesToEmitFile()
creates a new MCContext to pass to Target::createAsmPrinter.  Insights
welcome.  :)]

Perhaps there is a better way of doing this that's officially supported.
Is this model of function processing supported at all?  If not, has
anyone thought about how we might support it?  I think this is a model
that is not all that uncommon so I would hope we could figure out a
reasonable solution.

Thanks!

                     -Dave

Dear David,

Dumb question: Would it be easier to write an LLVM tool that took the 
large IR file and divided it into several smaller IR files that could 
then be code-generated separately?

-- John T.


David Greene wrote:
> We process very large programs and it is not unusual for the IR for some
> compilation unit to exceed system memory.  With some hacking in LLVM 2.5
> I was able to coax LLVM to generate asm for each functioin as it was
> processed and then completely forget about it (i.e. delete it) and move
> on to the next function.
>
> This required a bit of hackery.  I had to create two pass managers, one
> for the module and one for each function.  I called doInitialization on
> the module pass manager to convince the AsmPrinter to dump out global
> information needed at the top of the asm file.
>
> Then I would run each function though the function pass manager as they
> were encountered.  I could construct, run and destroy the function pass
> manager for each function.
>
> Finally I would run the module pass manager and doFinalization on the
> module pass manager to dump out the stuff needed at the end of the asm
> file.
>
> This was not at all pretty but I could at least get it to work.
>
> I'm in the last stages of upgrading to LLVM 2.7 and I'm finding
that the
> new MCStreamer system is making this sort of model much more difficult.
> MCContexts are somehow shared between the separate module and function
> pass managers in ways I don't understand.  I think that somehow the
> LLVMTargetMachine is shared between the pass managers.
>
> I think this sort of sharing is actually correct since we want the
> MCContext for the function pass manager to be the same as the one for
> the module pass manager.  That way globals, etc. have consistent
> representations in both.  Unfortunately, when the first function pass
> manager gets deleted, it destroys the MCContext alonmg with it.
> Actually, the AsmPrinter destroys the MCContext through a reference (!)
> which makes things all the more confusing.  When the module pass manager
> comes along and tries of access an MCSection, everything blows up.
>
> I had to re-add addAssemblyEmitter to LLVMTargetMachine to try to make
> this work.  It essentially takes the AsmPrinter creation bits from
> addPassesToEmitFile and packages them into a separate function.
>
> [As a side note, I don't understand how the two AsmPrinters (one for
the
> module pass manager, one for the function pass manager) end up pointing
> to the same MCContext as LLVMTargetMachine::addPassesToEmitFile()
> creates a new MCContext to pass to Target::createAsmPrinter.  Insights
> welcome.  :)]
>
> Perhaps there is a better way of doing this that's officially
supported.
> Is this model of function processing supported at all?  If not, has
> anyone thought about how we might support it?  I think this is a model
> that is not all that uncommon so I would hope we could figure out a
> reasonable solution.
>
> Thanks!
>
>                      -Dave
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

John Criswell <criswell at illinois.edu> writes:
> Dear David,
>
> Dumb question: Would it be easier to write an LLVM tool that took the 
> large IR file and divided it into several smaller IR files that could 
> then be code-generated separately?
Not a dumb question.  The problem is that we can't actually create the
entire LLVM IR to dump it to a file.  It's too big.  Instead, what we
did with 2.5 was create the module-level data (globals, etc.)  and then
handled each function separately: translate to LLVM IR, optimize, print
asm and delete.  We never had LLVM IR for the entire translation unit at
any time so we could not create an LLVM IR file if we wanted to.  I did
hack in a special mode to hold all the LLVM IR so we could dump it out.
But this was for debugging purposes and wouldn't work with some customer
codes which are very large.

                              -Dave

David Greene <dag at cray.com> writes:
> Perhaps there is a better way of doing this that's officially
supported.
> Is this model of function processing supported at all?  If not, has
> anyone thought about how we might support it?  I think this is a model
> that is not all that uncommon so I would hope we could figure out a
> reasonable solution.
After looking at the implementations for opt and llc, I think I can make
this work.  Apparently there is much better separation of concerns (asm
vs. dwarf generation, for example) that makes this a bit easier than
before.  In 2.5, the AsmPrinter made all sorts of assumptions about
dwarf information being available, which required some trickery for
module-level doInitialization and doFinalization, which is what led to
the separate module-/function-level asm printers.

Now it appears that with 2.7 I can go back to a single asm printer and
just invoke it on each function as I get them.  This seems to work on a
small set of testcases.

So it appears my troubles were in trying to adapt a hack for 2.5 to 2.7
when we don't need the hack at all.

                              -Dave