llvm dev - Jun 2019 - [RFC] Expressing preserved-relations between passes from different modules

Any comments at all on this? Chandler perhaps?

I've since dug a bit further, and it seems like the template-based
solution wouldn't work anyway because DLL loading on Windows can't do
the required commoning. So the general approach taken in
https://reviews.llvm.org/D62802 seems to be the only technically
viable path forward, though it would still be good to get an outside
look at the concrete details.

Thanks,
Nicolai

On Mon, Jun 3, 2019 at 10:35 AM Nicolai Hähnle-Montoro
<nhaehnle at gmail.com> wrote:
>
> Hi all,
>
> re-upping and renaming this thread to get some attention, as we'd like
> some feedback on a change that affects the future design direction.
>
> The concrete problem: We would like to mark the LCSSA pass as
> preserving the StackProtector pass in order to be able to properly
> express all dependencies we have. Unfortunately, those passes are
> defined in different modules that do not (and should not) depend on
> each other, which causes linking to fail.
>
> Our proposal is to introduce a collection of cross-module pass IDs in
> the core module (lib/IR) to work around such issues. This is done by
> the RFC patch at https://reviews.llvm.org/D62802.
>
> The only potential alternative I can see would be to generate the IDs
> using some kind of template magic and then rely on (runtime) linker
> magic to common the generated symbols. That feels like a bit too much
> magic for a fundamentally simple problem to me, hence the more
> low-tech proposal in the linked Phabricator review.
>
> Please let us know your thoughts!
>
> Thanks,
> Nicolai
>
> On Tue, May 14, 2019 at 3:55 PM Ryan Taylor via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Some background: We have an issue with in loop values being correctly
marked uniform but the out of loop uses can be non-uniform. Currently the out of
loop users are not marked as divergent because the in loop value is uniform
inside the loop. We have gotten around this problem for the moment by applying
LCSSA which inserts a PHI in the loop exit for the in loop uniform value that
allows the divergent info to be passed onto isel.
> >
> > The LCSSA is being inserted into XXXDAGToDAGISel class but this causes
a pass scheduling conflict with StackProtector. So what we've done is try to
preserve StackProtector in LCSSA, the issue is that the linker fails in Debug
build (but not Release build).
> >
> > StackProtector lies in CodeGen while LCSSA lies in Tranforms/Utils.
> >
> > Matt had mentioned to me that you can't refer to a preserved pass
if it's not on the same library and that it might make sense to move the
transform IDs into a separate library. We need a way to mark transforms/analyses
as preserved without depending on linking the transform itself.
> >
> > Thanks,
> >
> > Ryan
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> Lerne, wie die Welt wirklich ist,
> aber vergiss niemals, wie sie sein sollte.


-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.

On 6/6/19 3:27 AM, Nicolai Hähnle-Montoro via llvm-dev
wrote:
> Any comments at all on this? Chandler perhaps?
>
> I've since dug a bit further, and it seems like the template-based
> solution wouldn't work anyway because DLL loading on Windows can't
do
> the required commoning. So the general approach taken in
> https://reviews.llvm.org/D62802 seems to be the only technically
> viable path forward, though it would still be good to get an outside
> look at the concrete details.
>
> Thanks,
> Nicolai
>
> On Mon, Jun 3, 2019 at 10:35 AM Nicolai Hähnle-Montoro
> <nhaehnle at gmail.com> wrote:
>> Hi all,
>>
>> re-upping and renaming this thread to get some attention, as we'd
like
>> some feedback on a change that affects the future design direction.
>>
>> The concrete problem: We would like to mark the LCSSA pass as
>> preserving the StackProtector pass in order to be able to properly
>> express all dependencies we have. Unfortunately, those passes are
>> defined in different modules that do not (and should not) depend on
>> each other, which causes linking to fail.

You say "should not", but I'm not sure that's clear. If LCSSA
has
knowledge of the StackProtector, then there is a legitimate dependence. 
Either we express that directly, or alternatively, we abstract the 
relevant properties to create an abstract dependence. For example, we 
have the setPreservesCFG/CFGOnly property pair. In general, I think that 
we should have more of these abstract properties. That having been said, 
if the orthogonality is very specific, then a specific dependence may be 
appropriate.

  -Hal

>>
>> Our proposal is to introduce a collection of cross-module pass IDs in
>> the core module (lib/IR) to work around such issues. This is done by
>> the RFC patch at https://reviews.llvm.org/D62802.
>>
>> The only potential alternative I can see would be to generate the IDs
>> using some kind of template magic and then rely on (runtime) linker
>> magic to common the generated symbols. That feels like a bit too much
>> magic for a fundamentally simple problem to me, hence the more
>> low-tech proposal in the linked Phabricator review.
>>
>> Please let us know your thoughts!
>>
>> Thanks,
>> Nicolai
>>
>> On Tue, May 14, 2019 at 3:55 PM Ryan Taylor via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>>> Some background: We have an issue with in loop values being
correctly marked uniform but the out of loop uses can be non-uniform. Currently
the out of loop users are not marked as divergent because the in loop value is
uniform inside the loop. We have gotten around this problem for the moment by
applying LCSSA which inserts a PHI in the loop exit for the in loop uniform
value that allows the divergent info to be passed onto isel.
>>>
>>> The LCSSA is being inserted into XXXDAGToDAGISel class but this
causes a pass scheduling conflict with StackProtector. So what we've done is
try to preserve StackProtector in LCSSA, the issue is that the linker fails in
Debug build (but not Release build).
>>>
>>> StackProtector lies in CodeGen while LCSSA lies in Tranforms/Utils.
>>>
>>> Matt had mentioned to me that you can't refer to a preserved
pass if it's not on the same library and that it might make sense to move
the transform IDs into a separate library. We need a way to mark
transforms/analyses as preserved without depending on linking the transform
itself.
>>>
>>> Thanks,
>>>
>>> Ryan
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>> --
>> Lerne, wie die Welt wirklich ist,
>> aber vergiss niemals, wie sie sein sollte.
>
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

New week, new attempt at a ping...?

On Thu, Jun 6, 2019 at 10:27 AM Nicolai Hähnle-Montoro
<nhaehnle at gmail.com> wrote:
>
> Any comments at all on this? Chandler perhaps?
>
> I've since dug a bit further, and it seems like the template-based
> solution wouldn't work anyway because DLL loading on Windows can't
do
> the required commoning. So the general approach taken in
> https://reviews.llvm.org/D62802 seems to be the only technically
> viable path forward, though it would still be good to get an outside
> look at the concrete details.
>
> Thanks,
> Nicolai
>
> On Mon, Jun 3, 2019 at 10:35 AM Nicolai Hähnle-Montoro
> <nhaehnle at gmail.com> wrote:
> >
> > Hi all,
> >
> > re-upping and renaming this thread to get some attention, as we'd
like
> > some feedback on a change that affects the future design direction.
> >
> > The concrete problem: We would like to mark the LCSSA pass as
> > preserving the StackProtector pass in order to be able to properly
> > express all dependencies we have. Unfortunately, those passes are
> > defined in different modules that do not (and should not) depend on
> > each other, which causes linking to fail.
> >
> > Our proposal is to introduce a collection of cross-module pass IDs in
> > the core module (lib/IR) to work around such issues. This is done by
> > the RFC patch at https://reviews.llvm.org/D62802.
> >
> > The only potential alternative I can see would be to generate the IDs
> > using some kind of template magic and then rely on (runtime) linker
> > magic to common the generated symbols. That feels like a bit too much
> > magic for a fundamentally simple problem to me, hence the more
> > low-tech proposal in the linked Phabricator review.
> >
> > Please let us know your thoughts!
> >
> > Thanks,
> > Nicolai
> >
> > On Tue, May 14, 2019 at 3:55 PM Ryan Taylor via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> > >
> > > Some background: We have an issue with in loop values being
correctly marked uniform but the out of loop uses can be non-uniform. Currently
the out of loop users are not marked as divergent because the in loop value is
uniform inside the loop. We have gotten around this problem for the moment by
applying LCSSA which inserts a PHI in the loop exit for the in loop uniform
value that allows the divergent info to be passed onto isel.
> > >
> > > The LCSSA is being inserted into XXXDAGToDAGISel class but this
causes a pass scheduling conflict with StackProtector. So what we've done is
try to preserve StackProtector in LCSSA, the issue is that the linker fails in
Debug build (but not Release build).
> > >
> > > StackProtector lies in CodeGen while LCSSA lies in
Tranforms/Utils.
> > >
> > > Matt had mentioned to me that you can't refer to a preserved
pass if it's not on the same library and that it might make sense to move
the transform IDs into a separate library. We need a way to mark
transforms/analyses as preserved without depending on linking the transform
itself.
> > >
> > > Thanks,
> > >
> > > Ryan
> > >
> > >
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > llvm-dev at lists.llvm.org
> > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > --
> > Lerne, wie die Welt wirklich ist,
> > aber vergiss niemals, wie sie sein sollte.
>
>
>
> --
> Lerne, wie die Welt wirklich ist,
> aber vergiss niemals, wie sie sein sollte.


-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.

Huh, Gmail failed at threading, causing me to miss this response. Anyway:

On Thu, Jun 6, 2019 at 5:06 PM Finkel, Hal J. <hfinkel at anl.gov>
wrote:
> On 6/6/19 3:27 AM, Nicolai Hähnle-Montoro via llvm-dev wrote:
> > Any comments at all on this? Chandler perhaps?
> >
> > I've since dug a bit further, and it seems like the template-based
> > solution wouldn't work anyway because DLL loading on Windows
can't do
> > the required commoning. So the general approach taken in
> > https://reviews.llvm.org/D62802 seems to be the only technically
> > viable path forward, though it would still be good to get an outside
> > look at the concrete details.
> >
> > Thanks,
> > Nicolai
> >
> > On Mon, Jun 3, 2019 at 10:35 AM Nicolai Hähnle-Montoro
> > <nhaehnle at gmail.com> wrote:
> >> Hi all,
> >>
> >> re-upping and renaming this thread to get some attention, as
we'd like
> >> some feedback on a change that affects the future design
direction.
> >>
> >> The concrete problem: We would like to mark the LCSSA pass as
> >> preserving the StackProtector pass in order to be able to properly
> >> express all dependencies we have. Unfortunately, those passes are
> >> defined in different modules that do not (and should not) depend
on
> >> each other, which causes linking to fail.
>
>
> You say "should not", but I'm not sure that's clear. If
LCSSA has
> knowledge of the StackProtector, then there is a legitimate dependence.
> Either we express that directly, or alternatively, we abstract the
> relevant properties to create an abstract dependence. For example, we
> have the setPreservesCFG/CFGOnly property pair. In general, I think that
> we should have more of these abstract properties. That having been said,
> if the orthogonality is very specific, then a specific dependence may be
> appropriate.
There is mutual knowledge between the passes, yes. However:

1. Does that really mean that there is a _dependence_? And what kind
of dependence are we talking about anyway?
2. Even if there is a dependence, why should a linker be the correct
way of expressing it?

More to the point, what are you really trying to achieve here?

To the "abstract property" example, I don't think the analogy with
the
CFG quite holds because that is about analyses, unlike our situation.
In our use case, there are two passes modifying the IR to establish
what one might think of as different "normal forms", and we want to
express the fact that those are compatible and can both be established
simultaneously. So they are conceptually quite different things.

Note that this directly interacts with the linking question. If you
have two passes each establishing a kind of normal form, and they are
compatible with each other, then beyond the minimal change I've put on
Phabricator which is the bare minimum to make our use case work, you
quite likely want to say that they preserve each other in many cases,
in which case the linker requirement would cause a cyclical dependency
that breaks everything.

As an additional food for thought, I've also started to wonder if we'd
even be able to express this kind of dependency in the non-legacy pass
manager. Is this something we _want_ to be able to express in the pass
manager?

Cheers,
Nicolai


>
>   -Hal
>
>
> >>
> >> Our proposal is to introduce a collection of cross-module pass IDs
in
> >> the core module (lib/IR) to work around such issues. This is done
by
> >> the RFC patch at https://reviews.llvm.org/D62802.
> >>
> >> The only potential alternative I can see would be to generate the
IDs
> >> using some kind of template magic and then rely on (runtime)
linker
> >> magic to common the generated symbols. That feels like a bit too
much
> >> magic for a fundamentally simple problem to me, hence the more
> >> low-tech proposal in the linked Phabricator review.
> >>
> >> Please let us know your thoughts!
> >>
> >> Thanks,
> >> Nicolai
> >>
> >> On Tue, May 14, 2019 at 3:55 PM Ryan Taylor via llvm-dev
> >> <llvm-dev at lists.llvm.org> wrote:
> >>> Some background: We have an issue with in loop values being
correctly marked uniform but the out of loop uses can be non-uniform. Currently
the out of loop users are not marked as divergent because the in loop value is
uniform inside the loop. We have gotten around this problem for the moment by
applying LCSSA which inserts a PHI in the loop exit for the in loop uniform
value that allows the divergent info to be passed onto isel.
> >>>
> >>> The LCSSA is being inserted into XXXDAGToDAGISel class but
this causes a pass scheduling conflict with StackProtector. So what we've
done is try to preserve StackProtector in LCSSA, the issue is that the linker
fails in Debug build (but not Release build).
> >>>
> >>> StackProtector lies in CodeGen while LCSSA lies in
Tranforms/Utils.
> >>>
> >>> Matt had mentioned to me that you can't refer to a
preserved pass if it's not on the same library and that it might make sense
to move the transform IDs into a separate library. We need a way to mark
transforms/analyses as preserved without depending on linking the transform
itself.
> >>>
> >>> Thanks,
> >>>
> >>> Ryan
> >>>
> >>>
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >>
> >> --
> >> Lerne, wie die Welt wirklich ist,
> >> aber vergiss niemals, wie sie sein sollte.
> >
> >
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>

-- 
Lerne, wie die Welt wirklich ist,
aber vergiss niemals, wie sie sein sollte.