llvm dev - May 2015 - [LLVMdev] (Possibly buggy?) doFinalization method behavior of FunctionPass

On 5/6/15 10:19 AM, Kuperstein, Michael M wrote:
>
> Hello Cristiano,
>
> I don’t think doFinalization() is really meant to be used this way.
>
My understanding is that doInitialization() and doFinalization() are 
designed specifically for modifying the LLVM IR (otherwise, why would a 
mutable reference to the Function be provided)?

If that is not the case, then there is either a bug in the code or a bug 
in the documentation.

That said, I agree with the suggestion of writing a ModulePass. Since 
the PassManager does not run FunctionPasses in parallel yet, there's 
little benefit to using them.  I have often found the limitations on 
FunctionPasses to not be worth the hassle.

Regards,

John Criswell
> Its purpose is to allow clean-up of internal data-structures used by 
> the pass itself, not to make additional changes to the module.
>
> One option would be to rewrite your pass as a ModulePass instead of a 
> FunctionPass, then iterating over the functions manually, and doing 
> the final clean-up once that’s done.
>
> Michael
>
> *From:*llvmdev-bounces at cs.uiuc.edu 
> [mailto:llvmdev-bounces at cs.uiuc.edu] *On Behalf Of *Cristianno Martins
> *Sent:* Wednesday, May 06, 2015 03:20
> *To:* Lista LLVM-dev
> *Subject:* [LLVMdev] (Possibly buggy?) doFinalization method behavior 
> of FunctionPass
>
> Hello there,
>
> I'm writing some LLVM passes, and just ran into an interesting 
> situation: now, I don't know if I misunderstood the way doFinalization 
> is supposed to work, but I hope someone could help =)
>
> One of the transformations I wrote needed to replace some instructions 
> within the code, so I needed to clean up the code after the process 
> was completed. The pass basically swapped some function calls (from 
> the standard C library) with my own implementation of those functions. 
> Changing the code in this way, though, creates some dead code (like 
> those dead prototypes that are not being used anymore).
>
> I, then, implemented the "clean up" strategy overriding 
> doFinalization. Unfortunately, any modifications done to the module in 
> this method appears to be ignored by LLVM. I even dumped the module 
> directly from within the method, and could see that the modifications 
> were applied to that reference of the module, but the .bc file opt 
> wrote into does not retain these changes.
>
> Now, bear with me here: I know that other passes like DCE could be 
> used to clean the bytecode, but some of the code I implemented in 
> doFinalization actually needed to run only once, and necessarily after 
> the pass has finished: this is where I check to see if there is some 
> extra situation I need to address, optimize some of the replaced 
> instructions, and verify if any of the functions that I want to remove 
> had their addresses taken by any instruction.
>
> Also, doFinalization has a bool return type, but it doesn't appear to 
> have any different behavior if I return either value =/ (I assumed the 
> general idea would be "return true if the module was modified in any 
> way", like runOnFunction, but I couldn't find anything to support
that
> anywhere).
>
> Thus, am I wrong about how to use doFinalization? If so, is there any 
> way to guarantee running some code only once and only when a pass 
> already finished its job?
>
> Thanks in advance,
>
> Oh, and before I forget, this is the version of the opt I'm running:
>
> LLVM (http://llvm.org/):
>
>   LLVM version 3.7.0svn
>
>   DEBUG build with assertions.
>
>   Built May  4 2015 (00:18:21).
>
>   Default target: x86_64-apple-darwin14.3.0
>
>   Host CPU: sandybridge
>
>
> --
> Cristianno Martins
> PhD Student of Computer Science
> University of Campinas
> cmartins at ic.unicamp.br <mailto:cmartins at ic.unicamp.br>
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/cbd959cc/attachment.html>

I've always thought that the only guarantee is that doFinalization(Module
&M) runs after runOnFunction() was executed for all functions in M, and
there's no guarantee it runs *immediately* after.
That is, a PM may run a bunch of function passes over each function, and only
then call doFinazliation() for each pass. That means that, even though you get a
mutable reference to the module, the module you'll see is quite different
from what you may expect.

People more familiar with the pass managers - please correct me if I'm
wrong.

Michael

From: John Criswell [mailto:jtcriswel at gmail.com]
Sent: Wednesday, May 06, 2015 17:29
To: Kuperstein, Michael M; Cristianno Martins; Lista LLVM-dev
Subject: Re: [LLVMdev] (Possibly buggy?) doFinalization method behavior of
FunctionPass

On 5/6/15 10:19 AM, Kuperstein, Michael M wrote:
Hello Cristiano,

I don't think doFinalization() is really meant to be used this way.

My understanding is that doInitialization() and doFinalization() are designed
specifically for modifying the LLVM IR (otherwise, why would a mutable reference
to the Function be provided)?

If that is not the case, then there is either a bug in the code or a bug in the
documentation.

That said, I agree with the suggestion of writing a ModulePass.  Since the
PassManager does not run FunctionPasses in parallel yet, there's little
benefit to using them.  I have often found the limitations on FunctionPasses to
not be worth the hassle.

Regards,

John Criswell


Its purpose is to allow clean-up of internal data-structures used by the pass
itself, not to make additional changes to the module.

One option would be to rewrite your pass as a ModulePass instead of a
FunctionPass, then iterating over the functions manually, and doing the final
clean-up once that's done.

Michael

From: llvmdev-bounces at cs.uiuc.edu<mailto:llvmdev-bounces at
cs.uiuc.edu> [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Cristianno
Martins
Sent: Wednesday, May 06, 2015 03:20
To: Lista LLVM-dev
Subject: [LLVMdev] (Possibly buggy?) doFinalization method behavior of
FunctionPass

Hello there,

I'm writing some LLVM passes, and just ran into an interesting situation:
now, I don't know if I misunderstood the way doFinalization is supposed to
work, but I hope someone could help =)

One of the transformations I wrote needed to replace some instructions within
the code, so I needed to clean up the code after the process was completed. The
pass basically swapped some function calls (from the standard C library) with my
own implementation of those functions. Changing the code in this way, though,
creates some dead code (like those dead prototypes that are not being used
anymore).

I, then, implemented the "clean up" strategy overriding
doFinalization. Unfortunately, any modifications done to the module in this
method appears to be ignored by LLVM. I even dumped the module directly from
within the method, and could see that the modifications were applied to that
reference of the module, but the .bc file opt wrote into does not retain these
changes.

Now, bear with me here: I know that other passes like DCE could be used to clean
the bytecode, but some of the code I implemented in doFinalization actually
needed to run only once, and necessarily after the pass has finished: this is
where I check to see if there is some extra situation I need to address,
optimize some of the replaced instructions, and verify if any of the functions
that I want to remove had their addresses taken by any instruction.

Also, doFinalization has a bool return type, but it doesn't appear to have
any different behavior if I return either value =/ (I assumed the general idea
would be "return true if the module was modified in any way", like
runOnFunction, but I couldn't find anything to support that anywhere).

Thus, am I wrong about how to use doFinalization? If so, is there any way to
guarantee running some code only once and only when a pass already finished its
job?

Thanks in advance,

Oh, and before I forget, this is the version of the opt I'm running:

LLVM (http://llvm.org/):
  LLVM version 3.7.0svn
  DEBUG build with assertions.
  Built May  4 2015 (00:18:21).
  Default target: x86_64-apple-darwin14.3.0
  Host CPU: sandybridge

--
Cristianno Martins
PhD Student of Computer Science
University of Campinas
cmartins at ic.unicamp.br<mailto:cmartins at ic.unicamp.br>

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.




_______________________________________________

LLVM Developers mailing list

LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu

http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




--

John Criswell

Assistant Professor

Department of Computer Science, University of Rochester

http://www.cs.rochester.edu/u/criswell
---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/1891a8da/attachment.html>

On 5/6/15 11:15 AM, Kuperstein, Michael M wrote:
>
> I’ve always thought that the only guarantee is that 
> doFinalization(Module &M) runs after runOnFunction() was executed for 
> all functions in M, and there’s no guarantee it runs *immediately* after.
>
> That is, a PM may run a bunch of function passes over each function, 
> and only then call doFinazliation() for each pass. That means that, 
> even though you get a mutable reference to the module, the module 
> you’ll see is quite different from what you may expect.
>
Correct.  You're guaranteed that doFinalization() is run after your pass 
has been executed over all the functions.  There's no guarantees about 
what other passes are going to do either before or after 
doFinalization() is called.

Therefore, it's fine for doFinalization() to modify the Module.  You 
just have to be aware that other passes may change the Module later. 
That's why I asked whether there are any other passes executed after 
Cristianno's pass: they can (theoretically) add the function 
declarations back into the Module.

Regards,

John Criswell
> People more familiar with the pass managers – please correct me if I’m 
> wrong.
>
> Michael
>
> *From:*John Criswell [mailto:jtcriswel at gmail.com]
> *Sent:* Wednesday, May 06, 2015 17:29
> *To:* Kuperstein, Michael M; Cristianno Martins; Lista LLVM-dev
> *Subject:* Re: [LLVMdev] (Possibly buggy?) doFinalization method 
> behavior of FunctionPass
>
> On 5/6/15 10:19 AM, Kuperstein, Michael M wrote:
>
>     Hello Cristiano,
>
>     I don’t think doFinalization() is really meant to be used this way.
>
>
> My understanding is that doInitialization() and doFinalization() are 
> designed specifically for modifying the LLVM IR (otherwise, why would 
> a mutable reference to the Function be provided)?
>
> If that is not the case, then there is either a bug in the code or a 
> bug in the documentation.
>
> That said, I agree with the suggestion of writing a ModulePass.  Since 
> the PassManager does not run FunctionPasses in parallel yet, there's 
> little benefit to using them.  I have often found the limitations on 
> FunctionPasses to not be worth the hassle.
>
> Regards,
>
> John Criswell
>
>
> Its purpose is to allow clean-up of internal data-structures used by 
> the pass itself, not to make additional changes to the module.
>
> One option would be to rewrite your pass as a ModulePass instead of a 
> FunctionPass, then iterating over the functions manually, and doing 
> the final clean-up once that’s done.
>
> Michael
>
> *From:* llvmdev-bounces at cs.uiuc.edu 
> <mailto:llvmdev-bounces at cs.uiuc.edu> 
> [mailto:llvmdev-bounces at cs.uiuc.edu] *On Behalf Of *Cristianno Martins
> *Sent:* Wednesday, May 06, 2015 03:20
> *To:* Lista LLVM-dev
> *Subject:* [LLVMdev] (Possibly buggy?) doFinalization method behavior 
> of FunctionPass
>
> Hello there,
>
> I'm writing some LLVM passes, and just ran into an interesting 
> situation: now, I don't know if I misunderstood the way doFinalization 
> is supposed to work, but I hope someone could help =)
>
> One of the transformations I wrote needed to replace some instructions 
> within the code, so I needed to clean up the code after the process 
> was completed. The pass basically swapped some function calls (from 
> the standard C library) with my own implementation of those functions. 
> Changing the code in this way, though, creates some dead code (like 
> those dead prototypes that are not being used anymore).
>
> I, then, implemented the "clean up" strategy overriding 
> doFinalization. Unfortunately, any modifications done to the module in 
> this method appears to be ignored by LLVM. I even dumped the module 
> directly from within the method, and could see that the modifications 
> were applied to that reference of the module, but the .bc file opt 
> wrote into does not retain these changes.
>
> Now, bear with me here: I know that other passes like DCE could be 
> used to clean the bytecode, but some of the code I implemented in 
> doFinalization actually needed to run only once, and necessarily after 
> the pass has finished: this is where I check to see if there is some 
> extra situation I need to address, optimize some of the replaced 
> instructions, and verify if any of the functions that I want to remove 
> had their addresses taken by any instruction.
>
> Also, doFinalization has a bool return type, but it doesn't appear to 
> have any different behavior if I return either value =/ (I assumed the 
> general idea would be "return true if the module was modified in any 
> way", like runOnFunction, but I couldn't find anything to support
that
> anywhere).
>
> Thus, am I wrong about how to use doFinalization? If so, is there any 
> way to guarantee running some code only once and only when a pass 
> already finished its job?
>
> Thanks in advance,
>
> Oh, and before I forget, this is the version of the opt I'm running:
>
> LLVM (http://llvm.org/):
>
>   LLVM version 3.7.0svn
>
>   DEBUG build with assertions.
>
>   Built May  4 2015 (00:18:21).
>
>   Default target: x86_64-apple-darwin14.3.0
>
>   Host CPU: sandybridge
>
>
> --
> Cristianno Martins
> PhD Student of Computer Science
> University of Campinas
> cmartins at ic.unicamp.br <mailto:cmartins at ic.unicamp.br>
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu  <mailto:LLVMdev at cs.uiuc.edu>         
http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
>
> -- 
> John Criswell
> Assistant Professor
> Department of Computer Science, University of Rochester
> http://www.cs.rochester.edu/u/criswell
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/07829056/attachment.html>

Hello again,

First of all, thanks for all the answers =) they really helped a lot =D

*Have you verified that some other pass is not adding the function
declarations back in after your pass is executed (e.g., by using the
-debug-pass=Executions argument to see what passes run after your pass)?*

I considered that for a moment, but I realized that wouldn't be possible
for two reasons: I specifically ran my pass with opt level zero; and one of
the tasks I was doing was removing unused prototypes from the Module -- so,
why would any other doFinalization pass reintroduce those prototypes if
those functions were not being invoked anywhere? (PS: since this clean-up
happened inside my doFinalization, I'm assuming only other doFinalization
would be able to change the module after this point)


*Also, does your doFinalization() method make other changes that are
persistent?*


The main reason why I chose to use doFinalization was to check if any of
the functions I'm replacing still had any uses: since my pass replaced
every function call with my respective function calls, if any uses could
still be listed, that meant that some instruction must be taking the
address of that function, and I should treat those instructions at this
point. At the end, since I was already checking for uses, I ended up
removing the extra prototypes at this point, if they were still there.

I don’t think doFinalization() is really meant to be used this way.

Its purpose is to allow clean-up of internal data-structures used by the
pass itself, not to make additional changes to the module.


Then, I have a question: why should doFinalization need a Module& as
argument? I mean, since I shouldn't be able to modify it, and all data
structures I created to work my pass are known and (mostly) directly
accessible from within doFinalization, its argument would never be useful,
right?

One option would be to rewrite your pass as a ModulePass instead of a
FunctionPass, then iterating over the functions manually, and doing the
final clean-up once that’s done.

*That said, I agree with the suggestion of writing a ModulePass.  Since the
PassManager does not run FunctionPasses in parallel yet, there's little
benefit to using them.  I have often found the limitations on
FunctionPasses to not be worth the hassle.*

Well, I actually wouldn't mind using a ModulePass to begin with, but
unfortunately I had to rewrite my pass as a FunctionPass because part of
the job it does involve accessing the DominatorTree -- and, unfortunately,
I couldn't find a way to access it from inside a ModulePass. Even when I
try to force a dependence between my module pass and DominatorTree, I can't
get it to work because (if I'm not mistaken) module passes are hard-coded
to run before function passes =/

*My understanding is that doInitialization() and doFinalization() are
designed specifically for modifying the LLVM IR (otherwise, why would a
mutable reference to the Function be provided)?*


*If that is not the case, then there is either a bug in the code or a bug
in the documentation.*

Exactly like I thought =)


I’ve always thought that the only guarantee is that doFinalization(Module
&M) runs after runOnFunction() was executed for all functions in M, and
there’s no guarantee it runs *immediately* after.

That is, a PM may run a bunch of function passes over each function, and
only then call doFinazliation() for each pass. That means that, even though
you get a mutable reference to the module, the module you’ll see is quite
different from what you may expect.



People more familiar with the pass managers – please correct me if I’m
wrong.


I don't really mind at all if other passes run or even had their
doFinalization passes running in any order in relation to mine: the point
here is doFinalization is running (I know that because module.dump() is
being called from inside that function, and I can see the module containing
the modifications I do during the execution of that method), but the final
bc file (written by opt) does have the exactly same code that I get if I
dumped module at the beginning of my doFinalization (and they have
differences between them btw =)).




--
Cristianno Martins
PhD Student of Computer Science
University of Campinas
cmartins at ic.unicamp.br
<cristiannomartins at hotmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/2cb31db5/attachment.html>

> On May 6, 2015, at 7:28 AM, John Criswell <jtcriswel at gmail.com>
wrote:
> 
> My understanding is that doInitialization() and doFinalization() are
designed specifically for modifying the LLVM IR (otherwise, why would a mutable
reference to the Function be provided)?
As the person whose use case motivated added them, we definitely didn’t intend
them to be used for modifying IR.

—Owen
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/7fb97418/attachment.html>

Dear Owen,

First, to be sure we're on the same page, are we talking about 
doFinalization() in a FunctionPass or the doFinalization() method of 
another type of pass?  I've been assuming that we're talking about 
FunctionPasses.

Second, according to the documentation, FunctionPass::doInitialization() 
is allowed to modify the IR.  The documentation on doFinalization() does 
not say anything either way on the matter.

It sounds like there's a bug in the documentation.  Would you like me to 
file a bug report?

Regards,

John Criswell


On 5/6/15 12:45 PM, Owen Anderson wrote:
>
>> On May 6, 2015, at 7:28 AM, John Criswell <jtcriswel at gmail.com 
>> <mailto:jtcriswel at gmail.com>> wrote:
>>
>> My understanding is that doInitialization() and doFinalization() are 
>> designed specifically for modifying the LLVM IR (otherwise, why would 
>> a mutable reference to the Function be provided)?
>
> As the person whose use case motivated added them, we definitely 
> didn’t intend them to be used for modifying IR.
>
> —Owen

-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/5e593678/attachment.html>