llvm dev - Mar 2012 - [LLVMdev] VLIWPacketizerList: failing to schedule terminators

On Thu, Mar 29, 2012 at 02:57:27PM -0500, Sergei Larin
wrote:
> Tom,
> 
>    I do not have your call stack, but packetizer calls
> ScheduleDAGInstrs::buildSchedGraph to create dependency model. If this is
> the first time you use the new MI sched infrastructure (like your target
has
> not implemented misched yet) there might be some work needed to implement
> couple target hooks. isSchedulingBoundary is one of them. Also try to
> disable that assert and see what happens. It sounds strange, but in some
> cases that assert is overly restrictive.
Thanks, I disabled that assert, and I got a little farther.  I hit
some more assertion failures finalizeBundle, because the
instructions were still using virtual registers.  I'm not sure why
finalizeBundle doesn't currently handle virtual registers, maybe the
implementation isn't complete yet?

If I remove those asserts in finalizeBundle, I can get the program
to compile, and it has the VLIW bundles.

-Tom
> 
> Andy, 
> 
>   Those are just my guesses, but the issue looks very similar to our first
> experience with sched DAG constructor.
> 
> Sergei
> 
> --
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum.
> 
> 
> > -----Original Message-----
> > From: Tom Stellard [mailto:thomas.stellard at amd.com]
> > Sent: Thursday, March 29, 2012 2:23 PM
> > To: Sergei Larin
> > Cc: 'Anshuman Dasgupta'; llvmdev at cs.uiuc.edu
> > Subject: Re: VLIWPacketizerList: failing to schedule terminators
> > 
> > On Thu, Mar 29, 2012 at 01:50:58PM -0500, Sergei Larin wrote:
> > > Tom,
> > >
> > >   What is in your isSchedulingBoundary? If it contains isLabel
you
> > might
> > > need to disable that assert:
> > >
> > > assert(!MI->isTerminator() && !MI->isLabel()
&&
> > >            "Cannot schedule terminators or labels!");
> > >
> > > Sergei Larin
> > >
> > > --
> > > Qualcomm Innovation Center, Inc. is a member of Code Aurora
Forum.
> > >
> > >
> > 
> > I haven't implemented it, so it's still the default
implementation.
> > Though, the assertion failure is happening before the main packetizer
> > loop, so  it doesn't look like that function is ever called.
> > 
> > -Tom
> > 
> > 
> > > > -----Original Message-----
> > > > From: Tom Stellard [mailto:thomas.stellard at amd.com]
> > > > Sent: Thursday, March 29, 2012 11:01 AM
> > > > To: Anshuman Dasgupta; Sergei Larin
> > > > Cc: llvmdev at cs.uiuc.edu
> > > > Subject: VLIWPacketizerList: failing to schedule terminators
> > > >
> > > > Hi,
> > > >
> > > > I'm trying to use the VLIWPacketizerList to schedule
instructions
> > for
> > > > the R600 target, and I'm running into this assertion
failure:
> > > > ScheduleDAGInstrs.cpp:558: Cannot schedule terminators or
labels!
> > > >
> > > > I think I might not be using the VLIWPacketizerList class
> > correctly.
> > > > I've attached my code to this email.  Can anyone spot
what I'm
> > doing
> > > > wrong?
> > > >
> > > > Also, I had to add a LiveIntervals * parameter to the
constructor
> > of
> > > > this class in order to do pre-RA scheduling.
> > > >
> > > >
> > > > Thanks,
> > > > Tom
> > >
> > >
> 
> 
>

On Mar 29, 2012, at 1:18 PM, Tom Stellard <thomas.stellard at amd.com>
wrote:
> On Thu, Mar 29, 2012 at 02:57:27PM -0500, Sergei Larin wrote:
>> Tom,
>> 
>>   I do not have your call stack, but packetizer calls
>> ScheduleDAGInstrs::buildSchedGraph to create dependency model. If this
is
>> the first time you use the new MI sched infrastructure (like your
target has
>> not implemented misched yet) there might be some work needed to
implement
>> couple target hooks. isSchedulingBoundary is one of them. Also try to
>> disable that assert and see what happens. It sounds strange, but in
some
>> cases that assert is overly restrictive.
> 
> Thanks, I disabled that assert, and I got a little farther.  I hit
> some more assertion failures finalizeBundle, because the
> instructions were still using virtual registers.  I'm not sure why
> finalizeBundle doesn't currently handle virtual registers, maybe the
> implementation isn't complete yet?
> 
> If I remove those asserts in finalizeBundle, I can get the program
> to compile, and it has the VLIW bundles.
> 
> -Tom
> 
>> 
>> Andy, 
>> 
>>  Those are just my guesses, but the issue looks very similar to our
first
>> experience with sched DAG constructor.
>> 
>> Sergei
Tom,

The version of VLIWPacketizerList currently checked in was designed to work for
postRA scheduling. I want it to be adapted for use in preRA (called by
MachineScheduler), but Sergei and others are still engaged in that process.
Since nothing is checked in yet, you'll be running into the same issues as
Sergei.

To me, the definition of "isSchedulingBoundary" is "something the
DAG builder cannot handle". If you look at MachineScheduler::runOnFunction
it jumps through hoops to avoid exposing boundaries to buildSchedGraph().
VLIWPacketizer doesn't have that logic, so it only works if
isSchedulingBoundary returns false. Alternatively, you can disable the
buildSchedGraph assert and hope for the best--let's just say it's not
supported.

The MachineScheduler driver allows for two approaches to bundling:

1) Bundler operates on a larger scope than the scheduler (preferred):
isSchedulingBoundary is true for branches, calls, and other strange barriers.
Instructions can be reordered between the boundaries during schedule(), and the
bundler can operate across regions via exitRegion() and finishBlock(). You will
still have DAG edges from the instruction that a branch depends on to the branch
(captured by ExitSU), but you won't have any DAG edges crossing the
boundary.

2) Scheduler and bundler operate on the same scope (an entire block):
isSchedulingBoundary = false and all instructions must be correctly handled by
the DAG builder. DAG edges will be built for all instructions within the block.

---
Now, regarding bundle creation. The FinalizeMachineBundles pass will
"finalize" bundles for you after regalloc. You can enable that in your
target config. e.g.

bool MyTargetPassConfig::addFinalizeRegAlloc() {
    addPass(FinalizeMachineBundlesID);
}

This does not create bundles, rather it wraps bundles in an extra layer of
abstraction by copying all of the MachineOperands into a bundle header. I think
we have enough other bundle abstractions that this step could be eliminated, but
it's a bootstrapping mechanism that we rely on for now during postRA passes
only.

VLIWPacketizer calls finalizeBundle explicitly because it's running after
regalloc.

We do not want to see any Bundle instructions prior to regalloc. So a preRA
scheduler should not call finalizeBundles. It can create bundles simply by
setting the isInsideBundle flag on MachineInstrs. If you allow internal register
dependencies you would also need to set isInternalRead on those MachineOperands.

-Andy

On Thu, Mar 29, 2012 at 03:51:10PM -0700, Andrew Trick
wrote:
> 
> On Mar 29, 2012, at 1:18 PM, Tom Stellard <thomas.stellard at
amd.com> wrote:
> 
> > On Thu, Mar 29, 2012 at 02:57:27PM -0500, Sergei Larin wrote:
> >> Tom,
> >> 
> >>   I do not have your call stack, but packetizer calls
> >> ScheduleDAGInstrs::buildSchedGraph to create dependency model. If
this is
> >> the first time you use the new MI sched infrastructure (like your
target has
> >> not implemented misched yet) there might be some work needed to
implement
> >> couple target hooks. isSchedulingBoundary is one of them. Also try
to
> >> disable that assert and see what happens. It sounds strange, but
in some
> >> cases that assert is overly restrictive.
> > 
> > Thanks, I disabled that assert, and I got a little farther.  I hit
> > some more assertion failures finalizeBundle, because the
> > instructions were still using virtual registers.  I'm not sure why
> > finalizeBundle doesn't currently handle virtual registers, maybe
the
> > implementation isn't complete yet?
> > 
> > If I remove those asserts in finalizeBundle, I can get the program
> > to compile, and it has the VLIW bundles.
> > 
> > -Tom
> > 
> >> 
> >> Andy, 
> >> 
> >>  Those are just my guesses, but the issue looks very similar to
our first
> >> experience with sched DAG constructor.
> >> 
> >> Sergei
> 
> Tom,
> 
> The version of VLIWPacketizerList currently checked in was designed to work
for postRA scheduling. I want it to be adapted for use in preRA (called by
MachineScheduler), but Sergei and others are still engaged in that process.
Since nothing is checked in yet, you'll be running into the same issues as
Sergei.
Ok, so I think I'll do a postRA packetizer for now and them come back to my
preRA packetizer once VLIWPacketizerList can handle it.

The reason I wanted to do a preRA packetizer is because in the R600
target, the set of destination registers an instruction can write to
is determined by its spot within the bundle.  This means that either
after or during packetization, the backend needs to be able to change
the virtual register of an instruction to the class the corresponds to
its slot in the bundle.

> 
> To me, the definition of "isSchedulingBoundary" is
"something the DAG builder cannot handle". If you look at
MachineScheduler::runOnFunction it jumps through hoops to avoid exposing
boundaries to buildSchedGraph(). VLIWPacketizer doesn't have that logic, so
it only works if isSchedulingBoundary returns false. Alternatively, you can
disable the buildSchedGraph assert and hope for the best--let's just say
it's not supported.
> 
> The MachineScheduler driver allows for two approaches to bundling:
> 
> 1) Bundler operates on a larger scope than the scheduler (preferred):
isSchedulingBoundary is true for branches, calls, and other strange barriers.
Instructions can be reordered between the boundaries during schedule(), and the
bundler can operate across regions via exitRegion() and finishBlock(). You will
still have DAG edges from the instruction that a branch depends on to the branch
(captured by ExitSU), but you won't have any DAG edges crossing the
boundary.
> 
> 2) Scheduler and bundler operate on the same scope (an entire block):
isSchedulingBoundary = false and all instructions must be correctly handled by
the DAG builder. DAG edges will be built for all instructions within the block.
> 
> ---
> Now, regarding bundle creation. The FinalizeMachineBundles pass will
"finalize" bundles for you after regalloc. You can enable that in your
target config. e.g.
> 
> bool MyTargetPassConfig::addFinalizeRegAlloc() {
>     addPass(FinalizeMachineBundlesID);
> }
> 
> This does not create bundles, rather it wraps bundles in an extra layer of
abstraction by copying all of the MachineOperands into a bundle header. I think
we have enough other bundle abstractions that this step could be eliminated, but
it's a bootstrapping mechanism that we rely on for now during postRA passes
only.
> 
> VLIWPacketizer calls finalizeBundle explicitly because it's running
after regalloc.
> 
> We do not want to see any Bundle instructions prior to regalloc. So a preRA
scheduler should not call finalizeBundles. It can create bundles simply by
setting the isInsideBundle flag on MachineInstrs. If you allow internal register
dependencies you would also need to set isInternalRead on those MachineOperands.
>
Thanks, this information is really helpful.  One question, though:
If preRA bundles can't have Bundle instructions, then how does the
register allocator determine the boundaries between bundles?

-Tom