llvm dev - Nov 2016 - Loop Vectorize: Testing cost model driven transformations

Note: This is a continuation of a discussion over at
https://reviews.llvm.org/D26869.

Hi all,

In a discussion over on llvm-commits, we are debating how best to test loop
vectorization transformations that are guided by the cost model. The cost
model is currently used primarily for determining the vectorization and
interleave factors. Both of these parameters are easily overridden with
command line flags, which enables us to create target-independent tests.
Tests that rely on specific TTI hooks and instruction costs are placed
under the target-specific directories. Target-independent tests are great
because we don't have to replicate them for all targets.

But we can write new transformations (e.g., https://reviews.llvm.org/D26083)
that use the cost model to guide code generation in ways other than
selecting the vectorization and interleave factors. That is, the code we
generate can be different, depending on the target, even if we manually
specify vectorization and interleave factors.

So the question is, if we perform some new transformation based on cost
model results, how do we preserve the behavior of the existing
target-independent tests (that only specify vectorization and interleave
factors)? And also, how should we test the code that consumes the cost
model results but doesn't care about what those results are? Should the
tests be target-specific or target-independent?

There is some precedent for using TTI hooks to enable some optimizations,
with testing done in the target-specific directories. But if we expect an
optimization to be enabled for all targets does this make sense?

We currently have a "-force-target-instruction-cost" flag we can use
to get
consistency in the instruction costs. But as it's currently implemented, it
prevents us from testing with target-independent tests the code for
computing derived costs. For example, are we adding a scalarization
overhead to the cost of an instruction that can't be vectorized, are we
correctly scaling the cost of a predicated instruction by block
probability, etc. None of that logic is target-specific. Our current flag
just overrides the "final" cost of each instruction. But this is
somewhat
of an aside since our existing target-independent tests don't even use this
flag to guarantee instruction costs.

Some alternatives that we've discussed so far include:

   - Adding a new command line option to use the default TTI. This would
   allow us to use default values for TTI queries to ensure consistency across
   targets for transformations driven by the cost model. The existing and
   future target-independent tests would have to be updated to use the new
   flag. They also may still choose to manually specify vectorization and
   interleave factors to force vectorization and interleaving regardless of
   what the cost model would compute using the default TTI.
   - Adding a command line option to enable/disable each cost model driven
   transformation we add. The existing and future target-independent tests
   would have to be updated to explicitly disable each optimization (or in
   some way force predictable behavior across targets like we do now when
   setting the vectorization and interleave factors). All tests that test a
   cost model driven optimization would be placed under a target-specific
   directory.

Does anyone have any thoughts or suggestions?

Thanks!

-- Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161128/1ff35ece/attachment.html>

Hi Matt,

Thanks for summarizing the discussion here.
> On Nov 28, 2016, at 10:03 AM, Matthew Simpson <mssimpso at
codeaurora.org> wrote:
> 
> Note: This is a continuation of a discussion over at
https://reviews.llvm.org/D26869 <https://reviews.llvm.org/D26869>.
> 
> Hi all,
> 
> In a discussion over on llvm-commits, we are debating how best to test loop
vectorization transformations that are guided by the cost model. The cost model
is currently used primarily for determining the vectorization and interleave
factors. Both of these parameters are easily overridden with command line flags,
which enables us to create target-independent tests. Tests that rely on specific
TTI hooks and instruction costs are placed under the target-specific
directories. Target-independent tests are great because we don't have to
replicate them for all targets.
> 
> But we can write new transformations (e.g., https://reviews.llvm.org/D26083
<https://reviews.llvm.org/D26083>) that use the cost model to guide code
generation in ways other than selecting the vectorization and interleave
factors. That is, the code we generate can be different, depending on the
target, even if we manually specify vectorization and interleave factors.
> 
> So the question is, if we perform some new transformation based on cost
model results, how do we preserve the behavior of the existing
target-independent tests (that only specify vectorization and interleave
factors)? And also, how should we test the code that consumes the cost model
results but doesn't care about what those results are? Should the tests be
target-specific or target-independent?
> 
> There is some precedent for using TTI hooks to enable some optimizations,
with testing done in the target-specific directories. But if we expect an
optimization to be enabled for all targets does this make sense?
> 
> We currently have a "-force-target-instruction-cost" flag we can
use to get consistency in the instruction costs. But as it's currently
implemented, it prevents us from testing with target-independent tests the code
for computing derived costs. For example, are we adding a scalarization overhead
to the cost of an instruction that can't be vectorized, are we correctly
scaling the cost of a predicated instruction by block probability, etc. None of
that logic is target-specific. Our current flag just overrides the
"final" cost of each instruction. But this is somewhat of an aside
since our existing target-independent tests don't even use this flag to
guarantee instruction costs.
> 
> Some alternatives that we've discussed so far include:
> Adding a new command line option to use the default TTI. This would allow
us to use default values for TTI queries to ensure consistency across targets
for transformations driven by the cost model. The existing and future
target-independent tests would have to be updated to use the new flag. They also
may still choose to manually specify vectorization and interleave factors to
force vectorization and interleaving regardless of what the cost model would
compute using the default TTI.
Do we need a new (loop-vectorizer-specific) command line option for this?  Don’t
we get the default TTI if the target is unspecified in the test?

Adam
> Adding a command line option to enable/disable each cost model driven
transformation we add. The existing and future target-independent tests would
have to be updated to explicitly disable each optimization (or in some way force
predictable behavior across targets like we do now when setting the
vectorization and interleave factors). All tests that test a cost model driven
optimization would be placed under a target-specific directory.
> Does anyone have any thoughts or suggestions?
> 
> Thanks!
> 
> -- Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161129/2b2ee1c1/attachment.html>

On Tue, Nov 29, 2016 at 5:11 PM, Adam Nemet via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Do we need a new (loop-vectorizer-specific) command line option for this?
> Don’t we get the default TTI if the target is unspecified in the test?

I think you're right! It looks like I am getting the default TTI when the
target is left unspecified. I was assuming it would default to whatever the
host is, but this doesn't seem to be the case. I guess this is a non-issue
after all, as long as we don't specify a triple in the target-independent
tests. And it looks like Michael cleaned that all up in r283512.

Thanks!

-- Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161129/d9864c9b/attachment.html>