Ok any quick workaround to limit vectorization to 16-byte aligned 128-bit data then? All the memory copying done by ExpandUnalignedStore/ExpandUnalignedLoad is just too expensive. On Sat, Jul 20, 2013 at 12:52 PM, Arnold Schwaighofer < aschwaighofer at apple.com> wrote:> > On Jul 19, 2013, at 3:14 PM, Francois Pichet <pichet2000 at gmail.com> wrote: > > > > > What is the proper solution to disable auto-vectorization for unaligned > data? > > > > I have an out of tree target and I added this: > > > > bool OpusTargetLowering::allowsUnalignedMemoryAccesses(EVT VT, bool > *Fast) const { > > if (VT.isVector()) > > return false; > > .... > > } > > > > After that, I could see that vectorization is still done on unaligned > data except that llvm will copy the data back and forth from the source to > the top of the stack and work from there. This is very costly, I rather get > scalar operations. > > > > Then I tried to add: > > unsigned getMemoryOpCost(unsigned Opcode, Type *Src, > > unsigned Alignment, > > unsigned AddressSpace) const { > > if (Src->isVectorTy() && Alignment != 16) > > return 10000; // <== high number to try to avoid unaligned > load/store. > > return TargetTransformInfo::getMemoryOpCost(Opcode, Src, Alignment, > AddressSpace); > > } > > > > Except that this doesn't work because Alignment will always be 4 even > for data like: > > int data[16][16] __attribute__ ((aligned (16))), > > > > Because individual element are still 4-byte aligned. > > We will have to hook up some logic in the loop vectorizer that computes > the alignment of the vectorized version of the memory access so that we can > pass it to “getMemoryOpCost". Currently, as you have observed, we will just > pass the scalar loop’s memory access alignment which will be pessimistic. > > Instcombine will later replace the alignment to a stronger variant for > vectorized code but that is obviously to late for the cost model in the > vectorizer.-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130721/15b85cb6/attachment.html>
Arnold Schwaighofer
2013-Jul-21 17:16 UTC
[LLVMdev] Disable vectorization for unaligned data
No, I am afraid not without computing alignment based on the scalar code. In order to limit vectorization to 16-byte aligned data we need to know that data is 16-byte aligned. The way we vectorize we won’t know that until after we have vectorized. As you have observed we will pass “4” to getMemoryOpCost in the loop vectorizer (as that is the only thing that can be inferred from a consecutive scalar access like “aligned_ptr += 32bit”). scalar code -> estimate cost based on scalar instructions -> vectorize -> vectorized code -> ... -> instcombine (calls ComputeMaskedBits) which computes better alignment for pointer accesses like “aligned_ptr += 128bit”. I will have to work on this soon as ARM also has pretty inefficient unaligned vector loads. On Jul 21, 2013, at 9:29 AM, Francois Pichet <pichet2000 at gmail.com> wrote:> Ok any quick workaround to limit vectorization to 16-byte aligned 128-bit data then? > > All the memory copying done by ExpandUnalignedStore/ExpandUnalignedLoad is just too expensive. > > > On Sat, Jul 20, 2013 at 12:52 PM, Arnold Schwaighofer <aschwaighofer at apple.com> wrote: > > On Jul 19, 2013, at 3:14 PM, Francois Pichet <pichet2000 at gmail.com> wrote: > > > > > What is the proper solution to disable auto-vectorization for unaligned data? > > > > I have an out of tree target and I added this: > > > > bool OpusTargetLowering::allowsUnalignedMemoryAccesses(EVT VT, bool *Fast) const { > > if (VT.isVector()) > > return false; > > .... > > } > > > > After that, I could see that vectorization is still done on unaligned data except that llvm will copy the data back and forth from the source to the top of the stack and work from there. This is very costly, I rather get scalar operations. > > > > Then I tried to add: > > unsigned getMemoryOpCost(unsigned Opcode, Type *Src, > > unsigned Alignment, > > unsigned AddressSpace) const { > > if (Src->isVectorTy() && Alignment != 16) > > return 10000; // <== high number to try to avoid unaligned load/store. > > return TargetTransformInfo::getMemoryOpCost(Opcode, Src, Alignment, AddressSpace); > > } > > > > Except that this doesn't work because Alignment will always be 4 even for data like: > > int data[16][16] __attribute__ ((aligned (16))), > > > > Because individual element are still 4-byte aligned. > > We will have to hook up some logic in the loop vectorizer that computes the alignment of the vectorized version of the memory access so that we can pass it to “getMemoryOpCost". Currently, as you have observed, we will just pass the scalar loop’s memory access alignment which will be pessimistic. > > Instcombine will later replace the alignment to a stronger variant for vectorized code but that is obviously to late for the cost model in the vectorizer. > >
If I got you right, this is the classic case for loop peeling. I thought LLVM's vectorizer had something like that already in. On 21 July 2013 18:16, Arnold Schwaighofer <aschwaighofer at apple.com> wrote:> I will have to work on this soon as ARM also has pretty inefficient > unaligned vector loads. >NEON does support unaligned access via VLD*/VST*, what loads are you referring to? cheers, --renato -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130721/3b8188a3/attachment.html>