llvm dev - May 2015 - [LLVMdev] (Possibly buggy?) doFinalization method behavior of FunctionPass

On 5/6/15 11:15 AM, Kuperstein, Michael M wrote:
>
> I’ve always thought that the only guarantee is that 
> doFinalization(Module &M) runs after runOnFunction() was executed for 
> all functions in M, and there’s no guarantee it runs *immediately* after.
>
> That is, a PM may run a bunch of function passes over each function, 
> and only then call doFinazliation() for each pass. That means that, 
> even though you get a mutable reference to the module, the module 
> you’ll see is quite different from what you may expect.
>
Correct.  You're guaranteed that doFinalization() is run after your pass 
has been executed over all the functions.  There's no guarantees about 
what other passes are going to do either before or after 
doFinalization() is called.

Therefore, it's fine for doFinalization() to modify the Module.  You 
just have to be aware that other passes may change the Module later. 
That's why I asked whether there are any other passes executed after 
Cristianno's pass: they can (theoretically) add the function 
declarations back into the Module.

Regards,

John Criswell
> People more familiar with the pass managers – please correct me if I’m 
> wrong.
>
> Michael
>
> *From:*John Criswell [mailto:jtcriswel at gmail.com]
> *Sent:* Wednesday, May 06, 2015 17:29
> *To:* Kuperstein, Michael M; Cristianno Martins; Lista LLVM-dev
> *Subject:* Re: [LLVMdev] (Possibly buggy?) doFinalization method 
> behavior of FunctionPass
>
> On 5/6/15 10:19 AM, Kuperstein, Michael M wrote:
>
>     Hello Cristiano,
>
>     I don’t think doFinalization() is really meant to be used this way.
>
>
> My understanding is that doInitialization() and doFinalization() are 
> designed specifically for modifying the LLVM IR (otherwise, why would 
> a mutable reference to the Function be provided)?
>
> If that is not the case, then there is either a bug in the code or a 
> bug in the documentation.
>
> That said, I agree with the suggestion of writing a ModulePass.  Since 
> the PassManager does not run FunctionPasses in parallel yet, there's 
> little benefit to using them.  I have often found the limitations on 
> FunctionPasses to not be worth the hassle.
>
> Regards,
>
> John Criswell
>
>
> Its purpose is to allow clean-up of internal data-structures used by 
> the pass itself, not to make additional changes to the module.
>
> One option would be to rewrite your pass as a ModulePass instead of a 
> FunctionPass, then iterating over the functions manually, and doing 
> the final clean-up once that’s done.
>
> Michael
>
> *From:* llvmdev-bounces at cs.uiuc.edu 
> <mailto:llvmdev-bounces at cs.uiuc.edu> 
> [mailto:llvmdev-bounces at cs.uiuc.edu] *On Behalf Of *Cristianno Martins
> *Sent:* Wednesday, May 06, 2015 03:20
> *To:* Lista LLVM-dev
> *Subject:* [LLVMdev] (Possibly buggy?) doFinalization method behavior 
> of FunctionPass
>
> Hello there,
>
> I'm writing some LLVM passes, and just ran into an interesting 
> situation: now, I don't know if I misunderstood the way doFinalization 
> is supposed to work, but I hope someone could help =)
>
> One of the transformations I wrote needed to replace some instructions 
> within the code, so I needed to clean up the code after the process 
> was completed. The pass basically swapped some function calls (from 
> the standard C library) with my own implementation of those functions. 
> Changing the code in this way, though, creates some dead code (like 
> those dead prototypes that are not being used anymore).
>
> I, then, implemented the "clean up" strategy overriding 
> doFinalization. Unfortunately, any modifications done to the module in 
> this method appears to be ignored by LLVM. I even dumped the module 
> directly from within the method, and could see that the modifications 
> were applied to that reference of the module, but the .bc file opt 
> wrote into does not retain these changes.
>
> Now, bear with me here: I know that other passes like DCE could be 
> used to clean the bytecode, but some of the code I implemented in 
> doFinalization actually needed to run only once, and necessarily after 
> the pass has finished: this is where I check to see if there is some 
> extra situation I need to address, optimize some of the replaced 
> instructions, and verify if any of the functions that I want to remove 
> had their addresses taken by any instruction.
>
> Also, doFinalization has a bool return type, but it doesn't appear to 
> have any different behavior if I return either value =/ (I assumed the 
> general idea would be "return true if the module was modified in any 
> way", like runOnFunction, but I couldn't find anything to support
that
> anywhere).
>
> Thus, am I wrong about how to use doFinalization? If so, is there any 
> way to guarantee running some code only once and only when a pass 
> already finished its job?
>
> Thanks in advance,
>
> Oh, and before I forget, this is the version of the opt I'm running:
>
> LLVM (http://llvm.org/):
>
>   LLVM version 3.7.0svn
>
>   DEBUG build with assertions.
>
>   Built May  4 2015 (00:18:21).
>
>   Default target: x86_64-apple-darwin14.3.0
>
>   Host CPU: sandybridge
>
>
> --
> Cristianno Martins
> PhD Student of Computer Science
> University of Campinas
> cmartins at ic.unicamp.br <mailto:cmartins at ic.unicamp.br>
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu  <mailto:LLVMdev at cs.uiuc.edu>         
http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
>
> -- 
> John Criswell
> Assistant Professor
> Department of Computer Science, University of Rochester
> http://www.cs.rochester.edu/u/criswell
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/07829056/attachment.html>

It's not just a question of other passes changing the module between your
runOnFunction()-s and your doFinalization().
In theory, you may also have something like code-generation (or something else
with non-IR output) in the same PassManager as your pass. In this case,
doFinalization() will run too late for it to matter. So leaving a module in a
"half-baked" state until after doFinalization() runs seems ill-advised
to me.

Michael

From: John Criswell [mailto:jtcriswel at gmail.com]
Sent: Wednesday, May 06, 2015 19:02
To: Kuperstein, Michael M; Cristianno Martins; Lista LLVM-dev
Subject: Re: [LLVMdev] (Possibly buggy?) doFinalization method behavior of
FunctionPass

On 5/6/15 11:15 AM, Kuperstein, Michael M wrote:
I've always thought that the only guarantee is that doFinalization(Module
&M) runs after runOnFunction() was executed for all functions in M, and
there's no guarantee it runs *immediately* after.
That is, a PM may run a bunch of function passes over each function, and only
then call doFinazliation() for each pass. That means that, even though you get a
mutable reference to the module, the module you'll see is quite different
from what you may expect.

Correct.  You're guaranteed that doFinalization() is run after your pass has
been executed over all the functions.  There's no guarantees about what
other passes are going to do either before or after doFinalization() is called.

Therefore, it's fine for doFinalization() to modify the Module.  You just
have to be aware that other passes may change the Module later. That's why I
asked whether there are any other passes executed after Cristianno's pass:
they can (theoretically) add the function declarations back into the Module.

Regards,

John Criswell



People more familiar with the pass managers - please correct me if I'm
wrong.

Michael

From: John Criswell [mailto:jtcriswel at gmail.com]
Sent: Wednesday, May 06, 2015 17:29
To: Kuperstein, Michael M; Cristianno Martins; Lista LLVM-dev
Subject: Re: [LLVMdev] (Possibly buggy?) doFinalization method behavior of
FunctionPass

On 5/6/15 10:19 AM, Kuperstein, Michael M wrote:
Hello Cristiano,

I don't think doFinalization() is really meant to be used this way.

My understanding is that doInitialization() and doFinalization() are designed
specifically for modifying the LLVM IR (otherwise, why would a mutable reference
to the Function be provided)?

If that is not the case, then there is either a bug in the code or a bug in the
documentation.

That said, I agree with the suggestion of writing a ModulePass.  Since the
PassManager does not run FunctionPasses in parallel yet, there's little
benefit to using them.  I have often found the limitations on FunctionPasses to
not be worth the hassle.

Regards,

John Criswell

Its purpose is to allow clean-up of internal data-structures used by the pass
itself, not to make additional changes to the module.

One option would be to rewrite your pass as a ModulePass instead of a
FunctionPass, then iterating over the functions manually, and doing the final
clean-up once that's done.

Michael

From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at
cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Cristianno
Martins
Sent: Wednesday, May 06, 2015 03:20
To: Lista LLVM-dev
Subject: [LLVMdev] (Possibly buggy?) doFinalization method behavior of
FunctionPass

Hello there,

I'm writing some LLVM passes, and just ran into an interesting situation:
now, I don't know if I misunderstood the way doFinalization is supposed to
work, but I hope someone could help =)

One of the transformations I wrote needed to replace some instructions within
the code, so I needed to clean up the code after the process was completed. The
pass basically swapped some function calls (from the standard C library) with my
own implementation of those functions. Changing the code in this way, though,
creates some dead code (like those dead prototypes that are not being used
anymore).

I, then, implemented the "clean up" strategy overriding
doFinalization. Unfortunately, any modifications done to the module in this
method appears to be ignored by LLVM. I even dumped the module directly from
within the method, and could see that the modifications were applied to that
reference of the module, but the .bc file opt wrote into does not retain these
changes.

Now, bear with me here: I know that other passes like DCE could be used to clean
the bytecode, but some of the code I implemented in doFinalization actually
needed to run only once, and necessarily after the pass has finished: this is
where I check to see if there is some extra situation I need to address,
optimize some of the replaced instructions, and verify if any of the functions
that I want to remove had their addresses taken by any instruction.

Also, doFinalization has a bool return type, but it doesn't appear to have
any different behavior if I return either value =/ (I assumed the general idea
would be "return true if the module was modified in any way", like
runOnFunction, but I couldn't find anything to support that anywhere).

Thus, am I wrong about how to use doFinalization? If so, is there any way to
guarantee running some code only once and only when a pass already finished its
job?

Thanks in advance,

Oh, and before I forget, this is the version of the opt I'm running:

LLVM (http://llvm.org/):
  LLVM version 3.7.0svn
  DEBUG build with assertions.
  Built May  4 2015 (00:18:21).
  Default target: x86_64-apple-darwin14.3.0
  Host CPU: sandybridge

--
Cristianno Martins
PhD Student of Computer Science
University of Campinas
cmartins at ic.unicamp.br<mailto:cmartins at ic.unicamp.br>

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.



_______________________________________________

LLVM Developers mailing list

LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu

http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev



--

John Criswell

Assistant Professor

Department of Computer Science, University of Rochester

http://www.cs.rochester.edu/u/criswell

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.




--

John Criswell

Assistant Professor

Department of Computer Science, University of Rochester

http://www.cs.rochester.edu/u/criswell
---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150507/7cd7074b/attachment.html>

Hi again,

I think it’s wrongly documented. :-)


It could be, and I guess it probably is, but as I'm going to discuss below,
it should be interesting to have some function that bears this documented
doFinalization behavior =)


As a side note, I believe requiring transformation passes (as opposed to
requiring analysis passes) is also not-quite-supported.


I didn't know about that XD, and I think it makes sense (I don't even
have
to call getAnalysis from inside my FunctionPass, since I don't really mind
at all about any data flowing from the ModulePass), but it apparently
worked =/ I know I could just invoke both transformations from command line
(just by adding a different cmd option to opt), but, idk, it sounded kind
of weird having to write something like:

opt -load MyPasses.dylib -myPass -myPass_Extras <...>

since I always HAD TO have both being enabled together anyways.

It’s not just a question of other passes changing the module between your
runOnFunction()-s and your doFinalization().

In theory, you may also have something like code-generation (or something
else with non-IR output) in the same PassManager as your pass. In this
case, doFinalization() will run too late for it to matter. So leaving a
module in a “half-baked” state until after doFinalization() runs seems
ill-advised to me.


Well, to be fair, the main idea of having FunctionPasses in the first place
is that (please, correct me if I'm mistaken) opt could be running different
passes on different functions at the same time, so the compilation process
could be sped up. I know the parallel part of this is not yet implemented
on LLVM, but I think I can assume that my pass is going to work on a
function, and other passes will work over the same function after mine,
even before my pass can start again its work over the next function. In a
way, doesn't it mean that my pass keeps "half-baking" its work one
function
at a time until it finishes all of them for a given module?

This is actually why I think doFinalization should have a different
perspective between Module and "TheRestOfThe"-Passes: given that a
module
is the basic unit of compilation for opt, doFinalization is going to be
called as many times as runOnModule for every module you try to
transform/analyze on a ModulePass. This is not true for FunctionPass, for
example: if I have to do some work only once, and preferably after
finishing with all functions, I would have to insert some if clause into my
runOnFunction that checks a counter of functions and would effectively run
only if I already ran NumberOfFunctionsWithoutConsideringDefinitions times
-- which is hardly a good programming method. Besides, if what I assumed
above is really true (that there is no guarantee that your FunctionPass
finishes with the module before other FunctionPasses run over some of the
functions), even this strategy is worthless: the code could be changed by
other passes between the first and last execution of my pass runOnFunction
=/

Ok, but just for me not to be here only blabbing about "if we need to do
this only once"-case, I've looked into LLVM's code and found some
implementations of doFinalization that actually could hope the
modifications would be applied:

a. AsmPrinter: a MachineFunctionPass who's doFinalization is defined at
line 1004 of lib/CodeGen/AsmPrinter.cpp
b. Inliner: a CallGraphSCCPass, having doFinalization defined at
lib/Transforms/IPO/Inliner.cpp:621
c. BBPassManager: a FunctionPass, at lib/IR/LegacyPassManager.cpp:1344
d. FunctionPassManager: a Pass, at lib/IR/LegacyPassManager.cpp:1441

On the other hand, assuming the documentation is actually wrong, and
doFinalization shouldn't be allowed to change the module, I still
couldn't
see why exactly doFinalization functions have a Module& as argument (at
least the FunctionPass one), or a boolean return type for that matter; in
other words, I would guess the signature of this function is wrongly
defined anyways.

cheers,

--
Cristianno Martins
PhD Student of Computer Science
University of Campinas
cmartins at ic.unicamp.br
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150507/bcc8416d/attachment.html>