Dan Gohman
2009-May-20 23:55 UTC
[LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
On May 20, 2009, at 1:34 PM, Eli Friedman wrote:> On Wed, May 20, 2009 at 1:19 PM, Eli Friedman > <eli.friedman at gmail.com> wrote: > >> Per subject, this patch adding an additional pass to handle vector >> >> operations; the idea is that this allows removing the code from >> >> LegalizeDAG that handles illegal types, which should be a significant >> >> simplification. There are still some issues with this patch, but >> does >> >> the approach look sane?Can you explain why you chose the approach of using a new pass? I pictured removing LegalizeDAG's type legalization code would mostly consist of finding all the places that use TLI.getTypeAction and just deleting code for handling its Expand and Promote. Are you anticipating something more complicated? Dan
Eli Friedman
2009-May-21 00:26 UTC
[LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
On Wed, May 20, 2009 at 4:55 PM, Dan Gohman <gohman at apple.com> wrote:> Can you explain why you chose the approach of using a new pass? > I pictured removing LegalizeDAG's type legalization code would > mostly consist of finding all the places that use TLI.getTypeAction > and just deleting code for handling its Expand and Promote. Are you > anticipating something more complicated?Consider the case of an illegal operation (like division) on a <2 x i64> on x86-32. If we unroll it, the result has to go through type legalization. Therefore, if we want to remove the type legalization code from LegalizeDAG, we have to unroll vector operations first, then run a round of type legalization. (There are potential alternatives, like making LegalizeDAG call back into type legalization, but it gets messy.) Actually, I went ahead and tried writing the patch to remove type legalization from LegalizeDAG; I actually ran into another issue, which is the legalization of i32 UINT_TO_FP. We want to legalize it into an i64 UINT_TO_FP, but that requires type legalization. I'm not quite sure what to do about that; if I can't come up with a better solution, I'll force the x86 backend to custom-lower that case. -Eli
Eli Friedman
2009-May-21 04:14 UTC
[LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
On Wed, May 20, 2009 at 5:26 PM, Eli Friedman <eli.friedman at gmail.com> wrote:> On Wed, May 20, 2009 at 4:55 PM, Dan Gohman <gohman at apple.com> wrote: >> Can you explain why you chose the approach of using a new pass? >> I pictured removing LegalizeDAG's type legalization code would >> mostly consist of finding all the places that use TLI.getTypeAction >> and just deleting code for handling its Expand and Promote. Are you >> anticipating something more complicated? > > Consider the case of an illegal operation (like division) on a <2 x > i64> on x86-32. If we unroll it, the result has to go through type > legalization. Therefore, if we want to remove the type legalization > code from LegalizeDAG, we have to unroll vector operations first, then > run a round of type legalization. (There are potential alternatives, > like making LegalizeDAG call back into type legalization, but it gets > messy.) > > Actually, I went ahead and tried writing the patch to remove type > legalization from LegalizeDAG; I actually ran into another issue, > which is the legalization of i32 UINT_TO_FP. We want to legalize it > into an i64 UINT_TO_FP, but that requires type legalization. I'm not > quite sure what to do about that; if I can't come up with a better > solution, I'll force the x86 backend to custom-lower that case.Attached is a more complete version, including the simplification of LegalizeDAG and the fixes for UINT_TO_FP and FP_TO_UINT on x86. It passes regression tests except for one failure due to a missing implementation of legalization for EXTRACT_SUBVECTOR where both types are legal. -Eli -------------- next part -------------- Index: test/CodeGen/X86/widen_cast-4.ll ==================================================================--- test/CodeGen/X86/widen_cast-4.ll (revision 72159) +++ test/CodeGen/X86/widen_cast-4.ll (working copy) @@ -1,5 +1,5 @@ ; RUN: llvm-as < %s | llc -march=x86 -mattr=+sse42 -disable-mmx -o %t -f -; RUN: grep sarb %t | count 16 +; RUN: grep sarb %t | count 8 ; v8i8 that is widen to v16i8 then split ; FIXME: This is widen to v16i8 and split to 16 and we then rebuild the vector. Index: include/llvm/CodeGen/SelectionDAG.h ==================================================================--- include/llvm/CodeGen/SelectionDAG.h (revision 72159) +++ include/llvm/CodeGen/SelectionDAG.h (working copy) @@ -220,6 +220,13 @@ /// the graph. void Legalize(bool TypesNeedLegalizing, CodeGenOpt::Level OptLevel); + /// LegalizeVectors - This transforms the SelectionDAG into a SelectionDAG + /// that only uses vector operations supported by the target. + /// + /// Note that this is an involved process that may invalidate pointers into + /// the graph. + bool LegalizeVectors(); + /// RemoveDeadNodes - This method deletes all unreachable nodes in the /// SelectionDAG. void RemoveDeadNodes(); Index: lib/Target/X86/X86ISelLowering.cpp ==================================================================--- lib/Target/X86/X86ISelLowering.cpp (revision 72159) +++ lib/Target/X86/X86ISelLowering.cpp (working copy) @@ -117,15 +117,18 @@ setOperationAction(ISD::UINT_TO_FP , MVT::i32 , Promote); setOperationAction(ISD::UINT_TO_FP , MVT::i64 , Expand); } else { - if (!UseSoftFloat && !NoImplicitFloat && X86ScalarSSEf64) { + if (!UseSoftFloat && X86ScalarSSEf64) { // We have an impenetrably clever algorithm for ui64->double only. setOperationAction(ISD::UINT_TO_FP , MVT::i64 , Custom); - - // We have faster algorithm for ui32->single only. - setOperationAction(ISD::UINT_TO_FP , MVT::i32 , Custom); } else { - setOperationAction(ISD::UINT_TO_FP , MVT::i32 , Promote); + // Otherwise, let the legalizer turn this into an SINT_TO_FP + setOperationAction(ISD::UINT_TO_FP , MVT::i64 , Expand); } + if (!UseSoftFloat) { + // We have an algorithm for SSE2, and we turn this into a 64-bit + // FILD on other platforms. + setOperationAction(ISD::UINT_TO_FP , MVT::i32 , Custom); + } } // Promote i1/i8 SINT_TO_FP to larger SINT_TO_FP's, as X86 doesn't have @@ -176,15 +179,16 @@ if (Subtarget->is64Bit()) { setOperationAction(ISD::FP_TO_UINT , MVT::i64 , Expand); setOperationAction(ISD::FP_TO_UINT , MVT::i32 , Promote); - } else { + } else if (!UseSoftFloat) { if (X86ScalarSSEf32 && !Subtarget->hasSSE3()) // Expand FP_TO_UINT into a select. // FIXME: We would like to use a Custom expander here eventually to do // the optimal thing for SSE vs. the default expansion in the legalizer. - setOperationAction(ISD::FP_TO_UINT , MVT::i32 , Expand); + setOperationAction(ISD::FP_TO_UINT , MVT::i32, Expand); else - // With SSE3 we can use fisttpll to convert to a signed i64. - setOperationAction(ISD::FP_TO_UINT , MVT::i32 , Promote); + // With SSE3 we can use fisttpll to convert to a signed i64; without + // SSE, we're stuck with a fistpll. + setOperationAction(ISD::FP_TO_UINT , MVT::i32 , Custom); } // TODO: when we have SSE, these could be more efficient, by using movd/movq. @@ -4608,8 +4612,14 @@ SDValue Chain = DAG.getStore(DAG.getEntryNode(), dl, Op.getOperand(0), StackSlot, PseudoSourceValue::getFixedStack(SSFI), 0); + return BuildFILD(Op, SrcVT, Chain, StackSlot, DAG); +} +SDValue X86TargetLowering::BuildFILD(SDValue Op, MVT SrcVT, SDValue Chain, + SDValue StackSlot, + SelectionDAG &DAG) { // Build the FILD + DebugLoc dl = Op.getDebugLoc(); SDVTList Tys; bool useSSE = isScalarFPTypeInSSEReg(Op.getValueType()); if (useSSE) @@ -4792,36 +4802,54 @@ return SDValue(); return LowerUINT_TO_FP_i64(Op, DAG); - } else if (SrcVT == MVT::i32) { + } else if (SrcVT == MVT::i32 && X86ScalarSSEf64) { return LowerUINT_TO_FP_i32(Op, DAG); } - assert(0 && "Unknown UINT_TO_FP to lower!"); - return SDValue(); + assert(SrcVT == MVT::i32 && "Unknown UINT_TO_FP to lower!"); + + // Make a 64-bit buffer, and use it to build an FILD. + SDValue StackSlot = DAG.CreateStackTemporary(MVT::i64); + SDValue WordOff = DAG.getConstant(4, getPointerTy()); + SDValue OffsetSlot = DAG.getNode(ISD::ADD, dl, + getPointerTy(), StackSlot, WordOff); + SDValue Store1 = DAG.getStore(DAG.getEntryNode(), dl, Op, StackSlot, NULL, 0); + SDValue Store2 = DAG.getStore(Store1, dl, DAG.getConstant(0, MVT::i32), + OffsetSlot, NULL, 0); + return BuildFILD(Op, MVT::i64, Store2, StackSlot, DAG); } std::pair<SDValue,SDValue> X86TargetLowering:: -FP_TO_SINTHelper(SDValue Op, SelectionDAG &DAG) { +FP_TO_INTHelper(SDValue Op, SelectionDAG &DAG, bool IsSigned) { DebugLoc dl = Op.getDebugLoc(); - assert(Op.getValueType().getSimpleVT() <= MVT::i64 && - Op.getValueType().getSimpleVT() >= MVT::i16 && + + MVT DstTy = Op.getValueType(); + + if (!IsSigned) { + assert(DstTy == MVT::i32 && "Unexpected FP_TO_UINT"); + DstTy = MVT::i64; + } + + assert(DstTy.getSimpleVT() <= MVT::i64 && + DstTy.getSimpleVT() >= MVT::i16 && "Unknown FP_TO_SINT to lower!"); // These are really Legal. - if (Op.getValueType() == MVT::i32 && + if (DstTy == MVT::i32 && isScalarFPTypeInSSEReg(Op.getOperand(0).getValueType())) return std::make_pair(SDValue(), SDValue()); if (Subtarget->is64Bit() && - Op.getValueType() == MVT::i64 && + DstTy == MVT::i64 && Op.getOperand(0).getValueType() != MVT::f80) return std::make_pair(SDValue(), SDValue()); // We lower FP->sint64 into FISTP64, followed by a load, all to a temporary // stack slot. MachineFunction &MF = DAG.getMachineFunction(); - unsigned MemSize = Op.getValueType().getSizeInBits()/8; + unsigned MemSize = DstTy.getSizeInBits()/8; int SSFI = MF.getFrameInfo()->CreateStackObject(MemSize, MemSize); SDValue StackSlot = DAG.getFrameIndex(SSFI, getPointerTy()); + unsigned Opc; switch (Op.getValueType().getSimpleVT()) { default: assert(0 && "Invalid FP_TO_SINT to lower!"); @@ -4833,7 +4861,7 @@ SDValue Chain = DAG.getEntryNode(); SDValue Value = Op.getOperand(0); if (isScalarFPTypeInSSEReg(Op.getOperand(0).getValueType())) { - assert(Op.getValueType() == MVT::i64 && "Invalid FP_TO_SINT to lower!"); + assert(DstTy == MVT::i64 && "Invalid FP_TO_SINT to lower!"); Chain = DAG.getStore(Chain, dl, Value, StackSlot, PseudoSourceValue::getFixedStack(SSFI), 0); SDVTList Tys = DAG.getVTList(Op.getOperand(0).getValueType(), MVT::Other); @@ -4854,7 +4882,7 @@ } SDValue X86TargetLowering::LowerFP_TO_SINT(SDValue Op, SelectionDAG &DAG) { - std::pair<SDValue,SDValue> Vals = FP_TO_SINTHelper(Op, DAG); + std::pair<SDValue,SDValue> Vals = FP_TO_INTHelper(Op, DAG, true); SDValue FIST = Vals.first, StackSlot = Vals.second; if (FIST.getNode() == 0) return SDValue(); @@ -4863,6 +4891,16 @@ FIST, StackSlot, NULL, 0); } +SDValue X86TargetLowering::LowerFP_TO_UINT(SDValue Op, SelectionDAG &DAG) { + std::pair<SDValue,SDValue> Vals = FP_TO_INTHelper(Op, DAG, false); + SDValue FIST = Vals.first, StackSlot = Vals.second; + assert(FIST.getNode() && "Unexpected failure"); + + // Load the result. + return DAG.getLoad(Op.getValueType(), Op.getDebugLoc(), + FIST, StackSlot, NULL, 0); +} + SDValue X86TargetLowering::LowerFABS(SDValue Op, SelectionDAG &DAG) { DebugLoc dl = Op.getDebugLoc(); MVT VT = Op.getValueType(); @@ -6555,6 +6593,7 @@ case ISD::SINT_TO_FP: return LowerSINT_TO_FP(Op, DAG); case ISD::UINT_TO_FP: return LowerUINT_TO_FP(Op, DAG); case ISD::FP_TO_SINT: return LowerFP_TO_SINT(Op, DAG); + case ISD::FP_TO_UINT: return LowerFP_TO_UINT(Op, DAG); case ISD::FABS: return LowerFABS(Op, DAG); case ISD::FNEG: return LowerFNEG(Op, DAG); case ISD::FCOPYSIGN: return LowerFCOPYSIGN(Op, DAG); @@ -6626,7 +6665,8 @@ assert(false && "Do not know how to custom type legalize this operation!"); return; case ISD::FP_TO_SINT: { - std::pair<SDValue,SDValue> Vals = FP_TO_SINTHelper(SDValue(N, 0), DAG); + std::pair<SDValue,SDValue> Vals + FP_TO_INTHelper(SDValue(N, 0), DAG, true); SDValue FIST = Vals.first, StackSlot = Vals.second; if (FIST.getNode() != 0) { MVT VT = N->getValueType(0); Index: lib/Target/X86/X86ISelLowering.h ==================================================================--- lib/Target/X86/X86ISelLowering.h (revision 72159) +++ lib/Target/X86/X86ISelLowering.h (working copy) @@ -569,8 +569,8 @@ NameDecorationStyle NameDecorationForFORMAL_ARGUMENTS(SDValue Op); unsigned GetAlignedArgumentStackSize(unsigned StackSize, SelectionDAG &DAG); - std::pair<SDValue,SDValue> FP_TO_SINTHelper(SDValue Op, - SelectionDAG &DAG); + std::pair<SDValue,SDValue> FP_TO_INTHelper(SDValue Op, SelectionDAG &DAG, + bool isSigned); SDValue LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG); SDValue LowerVECTOR_SHUFFLE(SDValue Op, SelectionDAG &DAG); @@ -586,11 +586,16 @@ SDValue LowerGlobalTLSAddress(SDValue Op, SelectionDAG &DAG); SDValue LowerExternalSymbol(SDValue Op, SelectionDAG &DAG); SDValue LowerShift(SDValue Op, SelectionDAG &DAG); + SDValue BuildFILD(SDValue Op, MVT SrcVT, SDValue Chain, SDValue StackSlot, + SelectionDAG &DAG); + SDValue BuildFIST(SDValue Op, MVT DstVT, SDValue Chain, SDValue StackSlot, + SelectionDAG &DAG); SDValue LowerSINT_TO_FP(SDValue Op, SelectionDAG &DAG); SDValue LowerUINT_TO_FP(SDValue Op, SelectionDAG &DAG); SDValue LowerUINT_TO_FP_i64(SDValue Op, SelectionDAG &DAG); SDValue LowerUINT_TO_FP_i32(SDValue Op, SelectionDAG &DAG); SDValue LowerFP_TO_SINT(SDValue Op, SelectionDAG &DAG); + SDValue LowerFP_TO_UINT(SDValue Op, SelectionDAG &DAG); SDValue LowerFABS(SDValue Op, SelectionDAG &DAG); SDValue LowerFNEG(SDValue Op, SelectionDAG &DAG); SDValue LowerFCOPYSIGN(SDValue Op, SelectionDAG &DAG); Index: lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp ==================================================================--- lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp (revision 0) +++ lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp (revision 0) @@ -0,0 +1,276 @@ +//===-- LegalizeVectorOps.cpp - Implement SelectionDAG::LegalizeVectors ---===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// +// +// This file implements the SelectionDAG::LegalizeVectors method. +// +//===----------------------------------------------------------------------===// + +#include "llvm/CodeGen/SelectionDAG.h" +#include "llvm/Target/TargetLowering.h" +using namespace llvm; + +namespace { +class VectorLegalizer { + SelectionDAG& DAG; + TargetLowering& TLI; + SDValue UnrollVectorOp(SDValue Op); + SDValue UnrollVSETCC(SDValue Op); + SDValue ExpandFNEG(SDValue Op); + SDValue PromoteVectorOp(SDValue Op); + + public: + bool Run(); + VectorLegalizer(SelectionDAG& dag) : + DAG(dag), TLI(dag.getTargetLoweringInfo()) {} +}; + +bool VectorLegalizer::Run() { + bool Changed = false; + + // The vector legalizer is a relatively simple process because it doesn't + // need to legalize everything: it just needs to catch vector operations + // which might need to be unrolled. + DAG.AssignTopologicalOrder(); + for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(), + E = prior(DAG.allnodes_end()); I != next(E); ++I) { + bool HasVectorValue = false; + for (SDNode::value_iterator J = I->value_begin(); J != I->value_end(); ++J) + HasVectorValue |= J->isVector(); + if (!HasVectorValue) continue; + SDNode* Result = I; + switch (I->getOpcode()) { + default: + assert(I->getOpcode() > ISD::BUILTIN_OP_END && "Unexpected node!"); + break; + case ISD::UNDEF: + case ISD::FORMAL_ARGUMENTS: + case ISD::CALL: + case ISD::MERGE_VALUES: + case ISD::RET: + case ISD::VAARG: + case ISD::Register: + case ISD::INTRINSIC_WO_CHAIN: + case ISD::INTRINSIC_W_CHAIN: + case ISD::INTRINSIC_VOID: + case ISD::CopyToReg: + case ISD::CopyFromReg: + case ISD::AssertSext: + case ISD::AssertZext: + // Node cannot be illegal if types are legal + break; + case ISD::BUILD_VECTOR: + case ISD::INSERT_VECTOR_ELT: + case ISD::EXTRACT_VECTOR_ELT: + case ISD::CONCAT_VECTORS: + case ISD::EXTRACT_SUBVECTOR: + case ISD::VECTOR_SHUFFLE: + case ISD::SCALAR_TO_VECTOR: + case ISD::BIT_CONVERT: + case ISD::LOAD: + case ISD::STORE: + // These are intentionally not handled here; the point of this is to + // eliminate illegal operations that could potentially be unrolled to + // illegal types. Leaving them around can also help optimization. + break; + case ISD::ADD: + case ISD::SUB: + case ISD::MUL: + case ISD::SDIV: + case ISD::UDIV: + case ISD::SREM: + case ISD::UREM: + case ISD::FADD: + case ISD::FSUB: + case ISD::FMUL: + case ISD::FDIV: + case ISD::FREM: + case ISD::AND: + case ISD::OR: + case ISD::XOR: + case ISD::SHL: + case ISD::SRA: + case ISD::SRL: + case ISD::ROTL: + case ISD::ROTR: + case ISD::CTTZ: + case ISD::CTLZ: + case ISD::CTPOP: + case ISD::SELECT: + case ISD::SELECT_CC: + case ISD::VSETCC: + case ISD::ZERO_EXTEND: + case ISD::ANY_EXTEND: + case ISD::TRUNCATE: + case ISD::SIGN_EXTEND: + case ISD::SINT_TO_FP: + case ISD::UINT_TO_FP: + case ISD::FP_TO_SINT: + case ISD::FP_TO_UINT: + case ISD::FNEG: + case ISD::FABS: + case ISD::FSQRT: + case ISD::FSIN: + case ISD::FCOS: + case ISD::FPOWI: + case ISD::FPOW: + case ISD::FLOG: + case ISD::FLOG2: + case ISD::FLOG10: + case ISD::FEXP: + case ISD::FEXP2: + case ISD::FCEIL: + case ISD::FTRUNC: + case ISD::FRINT: + case ISD::FNEARBYINT: + case ISD::FFLOOR: + switch (TLI.getOperationAction(I->getOpcode(), I->getValueType(0))) { + case TargetLowering::Promote: + Result = PromoteVectorOp(SDValue(Result, 0)).getNode(); + break; + case TargetLowering::Legal: break; + case TargetLowering::Custom: { + SDValue Tmp1 = TLI.LowerOperation(SDValue(Result, 0), DAG); + if (Tmp1.getNode()) { + // FIXME: Should the returned value be recursively checked? + Result = Tmp1.getNode(); + break; + } + // FALL THROUGH + } + case TargetLowering::Expand: + // FIXME: Handle some special cases: FNEG->FSUB + if (I->getOpcode() == ISD::FNEG) + Result = ExpandFNEG(SDValue(Result, 0)).getNode(); + else if (I->getOpcode() == ISD::VSETCC) + Result = UnrollVSETCC(SDValue(Result, 0)).getNode(); + else + Result = UnrollVectorOp(SDValue(Result, 0)).getNode(); + break; + } + break; + } + if (&*I != Result) { + Changed = true; + DAG.ReplaceAllUsesWith(I, Result); + } + } + + // Remove dead nodes now. + DAG.RemoveDeadNodes(); + + return Changed; +} + +SDValue VectorLegalizer::PromoteVectorOp(SDValue Op) { + MVT VT = Op.getValueType(); + assert(Op.getNode()->getNumValues() == 1 && + "Can't promote a vector with multiple results!"); + MVT NVT = TLI.getTypeToPromoteTo(Op.getOpcode(), VT); + DebugLoc dl = Op.getDebugLoc(); + SmallVector<SDValue, 4> Operands(Op.getNumOperands()); + + for (unsigned j = 0; j != Op.getNumOperands(); ++j) { + if (Op.getOperand(j).getValueType().isVector()) + Operands[j] = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Op.getOperand(j)); + else + Operands[j] = Op.getOperand(j); + } + + Op = DAG.getNode(Op.getOpcode(), dl, NVT, &Operands[0], Operands.size()); + + return DAG.getNode(ISD::BIT_CONVERT, dl, VT, Op); +} + +SDValue VectorLegalizer::ExpandFNEG(SDValue Op) { + if (TLI.isOperationLegalOrCustom(ISD::FSUB, Op.getValueType())) { + SDValue Zero = DAG.getConstantFP(-0.0, Op.getValueType()); + return DAG.getNode(ISD::FSUB, Op.getDebugLoc(), Op.getValueType(), + Zero, Op.getOperand(0)); + } + return UnrollVectorOp(Op); +} + +SDValue VectorLegalizer::UnrollVSETCC(SDValue Op) { + MVT VT = Op.getValueType(); + unsigned NumElems = VT.getVectorNumElements(); + MVT EltVT = VT.getVectorElementType(); + SDValue LHS = Op.getOperand(0), RHS = Op.getOperand(1), CC = Op.getOperand(2); + MVT TmpEltVT = LHS.getValueType().getVectorElementType(); + DebugLoc dl = Op.getDebugLoc(); + SmallVector<SDValue, 8> Ops(NumElems); + for (unsigned i = 0; i < NumElems; ++i) { + SDValue LHSElem = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, LHS, + DAG.getIntPtrConstant(i)); + SDValue RHSElem = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, RHS, + DAG.getIntPtrConstant(i)); + Ops[i] = DAG.getNode(ISD::SETCC, dl, TLI.getSetCCResultType(TmpEltVT), + LHSElem, RHSElem, CC); + Ops[i] = DAG.getNode(ISD::SELECT, dl, EltVT, Ops[i], + DAG.getConstant(APInt::getAllOnesValue + (EltVT.getSizeInBits()), EltVT), + DAG.getConstant(0, EltVT)); + } + return DAG.getNode(ISD::BUILD_VECTOR, dl, VT, &Ops[0], NumElems); +} + +/// UnrollVectorOp - We know that the given vector has a legal type, however +/// the operation it performs is not legal, and the target has requested that +/// the operation be expanded. "Unroll" the vector, splitting out the scalars +/// and operating on each element individually. +SDValue VectorLegalizer::UnrollVectorOp(SDValue Op) { + MVT VT = Op.getValueType(); + assert(Op.getNode()->getNumValues() == 1 && + "Can't unroll a vector with multiple results!"); + unsigned NE = VT.getVectorNumElements(); + MVT EltVT = VT.getVectorElementType(); + DebugLoc dl = Op.getDebugLoc(); + + SmallVector<SDValue, 8> Scalars; + SmallVector<SDValue, 4> Operands(Op.getNumOperands()); + for (unsigned i = 0; i != NE; ++i) { + for (unsigned j = 0; j != Op.getNumOperands(); ++j) { + SDValue Operand = Op.getOperand(j); + MVT OperandVT = Operand.getValueType(); + if (OperandVT.isVector()) { + // A vector operand; extract a single element. + MVT OperandEltVT = OperandVT.getVectorElementType(); + Operands[j] = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, + OperandEltVT, + Operand, + DAG.getConstant(i, MVT::i32)); + } else { + // A scalar operand; just use it as is. + Operands[j] = Operand; + } + } + + switch (Op.getOpcode()) { + default: + Scalars.push_back(DAG.getNode(Op.getOpcode(), dl, EltVT, + &Operands[0], Operands.size())); + break; + case ISD::SHL: + case ISD::SRA: + case ISD::SRL: + case ISD::ROTL: + case ISD::ROTR: + Scalars.push_back(DAG.getNode(Op.getOpcode(), dl, EltVT, Operands[0], + DAG.getShiftAmountOperand(Operands[1]))); + break; + } + } + + return DAG.getNode(ISD::BUILD_VECTOR, dl, VT, &Scalars[0], Scalars.size()); +} + +} + +bool SelectionDAG::LegalizeVectors() { + return VectorLegalizer(*this).Run(); +} Index: lib/CodeGen/SelectionDAG/LegalizeDAG.cpp ==================================================================--- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp (revision 72159) +++ lib/CodeGen/SelectionDAG/LegalizeDAG.cpp (working copy) @@ -70,11 +70,6 @@ /// being legalized (which could lead to non-serialized call sequences). bool IsLegalizingCall; - /// IsLegalizingCallArguments - This member is used only for the purpose - /// of providing assert to check for LegalizeTypes because legalizing an - /// operation might introduce call nodes that might need type legalization. - bool IsLegalizingCallArgs; - enum LegalizeAction { Legal, // The target natively supports this operation. Promote, // This operation should be executed in a larger type. @@ -91,51 +86,12 @@ /// allows us to avoid legalizing the same thing more than once. DenseMap<SDValue, SDValue> LegalizedNodes; - /// PromotedNodes - For nodes that are below legal width, and that have more - /// than one use, this map indicates what promoted value to use. This allows - /// us to avoid promoting the same thing more than once. - DenseMap<SDValue, SDValue> PromotedNodes; - - /// ExpandedNodes - For nodes that need to be expanded this map indicates - /// which operands are the expanded version of the input. This allows - /// us to avoid expanding the same node more than once. - DenseMap<SDValue, std::pair<SDValue, SDValue> > ExpandedNodes; - - /// SplitNodes - For vector nodes that need to be split, this map indicates - /// which operands are the split version of the input. This allows us - /// to avoid splitting the same node more than once. - std::map<SDValue, std::pair<SDValue, SDValue> > SplitNodes; - - /// ScalarizedNodes - For nodes that need to be converted from vector types to - /// scalar types, this contains the mapping of ones we have already - /// processed to the result. - std::map<SDValue, SDValue> ScalarizedNodes; - - /// WidenNodes - For nodes that need to be widened from one vector type to - /// another, this contains the mapping of those that we have already widen. - /// This allows us to avoid widening more than once. - std::map<SDValue, SDValue> WidenNodes; - void AddLegalizedOperand(SDValue From, SDValue To) { LegalizedNodes.insert(std::make_pair(From, To)); // If someone requests legalization of the new node, return itself. if (From != To) LegalizedNodes.insert(std::make_pair(To, To)); } - void AddPromotedOperand(SDValue From, SDValue To) { - bool isNew = PromotedNodes.insert(std::make_pair(From, To)).second; - assert(isNew && "Got into the map somehow?"); - isNew = isNew; - // If someone requests legalization of the new node, return itself. - LegalizedNodes.insert(std::make_pair(To, To)); - } - void AddWidenedOperand(SDValue From, SDValue To) { - bool isNew = WidenNodes.insert(std::make_pair(From, To)).second; - assert(isNew && "Got into the map somehow?"); - isNew = isNew; - // If someone requests legalization of the new node, return itself. - LegalizedNodes.insert(std::make_pair(To, To)); - } public: explicit SelectionDAGLegalize(SelectionDAG &DAG, bool TypesNeedLegalizing, @@ -157,21 +113,11 @@ void LegalizeDAG(); private: - /// HandleOp - Legalize, Promote, or Expand the specified operand as - /// appropriate for its type. - void HandleOp(SDValue Op); - /// LegalizeOp - We know that the specified value has a legal type. /// Recursively ensure that the operands have legal types, then return the /// result. SDValue LegalizeOp(SDValue O); - /// UnrollVectorOp - We know that the given vector has a legal type, however - /// the operation it performs is not legal and is an operation that we have - /// no way of lowering. "Unroll" the vector, splitting out the scalars and - /// operating on each element individually. - SDValue UnrollVectorOp(SDValue O); - /// PerformInsertVectorEltInMemory - Some target cannot handle a variable /// insertion index for the INSERT_VECTOR_ELT instruction. In this case, it /// is necessary to spill the vector being inserted into to memory, perform @@ -179,94 +125,9 @@ SDValue PerformInsertVectorEltInMemory(SDValue Vec, SDValue Val, SDValue Idx, DebugLoc dl); - /// PromoteOp - Given an operation that produces a value in an invalid type, - /// promote it to compute the value into a larger type. The produced value - /// will have the correct bits for the low portion of the register, but no - /// guarantee is made about the top bits: it may be zero, sign-extended, or - /// garbage. - SDValue PromoteOp(SDValue O); - - /// ExpandOp - Expand the specified SDValue into its two component pieces - /// Lo&Hi. Note that the Op MUST be an expanded type. As a result of this, - /// the LegalizedNodes map is filled in for any results that are not expanded, - /// the ExpandedNodes map is filled in for any results that are expanded, and - /// the Lo/Hi values are returned. This applies to integer types and Vector - /// types. - void ExpandOp(SDValue O, SDValue &Lo, SDValue &Hi); - - /// WidenVectorOp - Widen a vector operation to a wider type given by WidenVT - /// (e.g., v3i32 to v4i32). The produced value will have the correct value - /// for the existing elements but no guarantee is made about the new elements - /// at the end of the vector: it may be zero, ones, or garbage. This is useful - /// when we have an instruction operating on an illegal vector type and we - /// want to widen it to do the computation on a legal wider vector type. - SDValue WidenVectorOp(SDValue Op, MVT WidenVT); - - /// SplitVectorOp - Given an operand of vector type, break it down into - /// two smaller values. - void SplitVectorOp(SDValue O, SDValue &Lo, SDValue &Hi); - - /// ScalarizeVectorOp - Given an operand of single-element vector type - /// (e.g. v1f32), convert it into the equivalent operation that returns a - /// scalar (e.g. f32) value. - SDValue ScalarizeVectorOp(SDValue O); - /// Useful 16 element vector type that is used to pass operands for widening. typedef SmallVector<SDValue, 16> SDValueVector; - /// LoadWidenVectorOp - Load a vector for a wider type. Returns true if - /// the LdChain contains a single load and false if it contains a token - /// factor for multiple loads. It takes - /// Result: location to return the result - /// LdChain: location to return the load chain - /// Op: load operation to widen - /// NVT: widen vector result type we want for the load - bool LoadWidenVectorOp(SDValue& Result, SDValue& LdChain, - SDValue Op, MVT NVT); - - /// Helper genWidenVectorLoads - Helper function to generate a set of - /// loads to load a vector with a resulting wider type. It takes - /// LdChain: list of chains for the load we have generated - /// Chain: incoming chain for the ld vector - /// BasePtr: base pointer to load from - /// SV: memory disambiguation source value - /// SVOffset: memory disambiugation offset - /// Alignment: alignment of the memory - /// isVolatile: volatile load - /// LdWidth: width of memory that we want to load - /// ResType: the wider result result type for the resulting loaded vector - SDValue genWidenVectorLoads(SDValueVector& LdChain, SDValue Chain, - SDValue BasePtr, const Value *SV, - int SVOffset, unsigned Alignment, - bool isVolatile, unsigned LdWidth, - MVT ResType, DebugLoc dl); - - /// StoreWidenVectorOp - Stores a widen vector into non widen memory - /// location. It takes - /// ST: store node that we want to replace - /// Chain: incoming store chain - /// BasePtr: base address of where we want to store into - SDValue StoreWidenVectorOp(StoreSDNode *ST, SDValue Chain, - SDValue BasePtr); - - /// Helper genWidenVectorStores - Helper function to generate a set of - /// stores to store a widen vector into non widen memory - // It takes - // StChain: list of chains for the stores we have generated - // Chain: incoming chain for the ld vector - // BasePtr: base pointer to load from - // SV: memory disambiguation source value - // SVOffset: memory disambiugation offset - // Alignment: alignment of the memory - // isVolatile: volatile lod - // ValOp: value to store - // StWidth: width of memory that we want to store - void genWidenVectorStores(SDValueVector& StChain, SDValue Chain, - SDValue BasePtr, const Value *SV, - int SVOffset, unsigned Alignment, - bool isVolatile, SDValue ValOp, - unsigned StWidth, DebugLoc dl); - /// ShuffleWithNarrowerEltType - Return a vector shuffle operation which /// performs the same shuffe in terms of order or result bytes, but on a type /// whose vector element type is narrower than the original shuffle type. @@ -278,19 +139,17 @@ bool LegalizeAllNodesNotLeadingTo(SDNode *N, SDNode *Dest, SmallPtrSet<SDNode*, 32> &NodesLeadingTo); - void LegalizeSetCCOperands(SDValue &LHS, SDValue &RHS, SDValue &CC, - DebugLoc dl); void LegalizeSetCCCondCode(MVT VT, SDValue &LHS, SDValue &RHS, SDValue &CC, DebugLoc dl); void LegalizeSetCC(MVT VT, SDValue &LHS, SDValue &RHS, SDValue &CC, DebugLoc dl) { - LegalizeSetCCOperands(LHS, RHS, CC, dl); + LHS = LegalizeOp(LHS); + RHS = LegalizeOp(RHS); LegalizeSetCCCondCode(VT, LHS, RHS, CC, dl); } SDValue ExpandLibCall(RTLIB::Libcall LC, SDNode *Node, bool isSigned, SDValue &Hi); - SDValue ExpandIntToFP(bool isSigned, MVT DestTy, SDValue Source, DebugLoc dl); SDValue EmitStackConvert(SDValue SrcOp, MVT SlotVT, MVT DestVT, DebugLoc dl); SDValue ExpandBUILD_VECTOR(SDNode *Node); @@ -306,12 +165,7 @@ SDValue ExpandBSWAP(SDValue Op, DebugLoc dl); SDValue ExpandBitCount(unsigned Opc, SDValue Op, DebugLoc dl); - bool ExpandShift(unsigned Opc, SDValue Op, SDValue Amt, - SDValue &Lo, SDValue &Hi, DebugLoc dl); - void ExpandShiftParts(unsigned NodeOp, SDValue Op, SDValue Amt, - SDValue &Lo, SDValue &Hi, DebugLoc dl); - SDValue ExpandEXTRACT_SUBVECTOR(SDValue Op); SDValue ExpandEXTRACT_VECTOR_ELT(SDValue Op); }; } @@ -360,7 +214,6 @@ void SelectionDAGLegalize::LegalizeDAG() { LastCALLSEQ_END = DAG.getEntryNode(); IsLegalizingCall = false; - IsLegalizingCallArgs = false; // The legalize process is inherently a bottom-up recursive process (users // legalize their uses before themselves). Given infinite stack space, we @@ -371,19 +224,14 @@ DAG.AssignTopologicalOrder(); for (SelectionDAG::allnodes_iterator I = DAG.allnodes_begin(), E = prior(DAG.allnodes_end()); I != next(E); ++I) - HandleOp(SDValue(I, 0)); + LegalizeOp(SDValue(I, 0)); // Finally, it's possible the root changed. Get the new root. SDValue OldRoot = DAG.getRoot(); assert(LegalizedNodes.count(OldRoot) && "Root didn't get legalized?"); DAG.setRoot(LegalizedNodes[OldRoot]); - ExpandedNodes.clear(); LegalizedNodes.clear(); - PromotedNodes.clear(); - SplitNodes.clear(); - ScalarizedNodes.clear(); - WidenNodes.clear(); // Remove dead nodes now. DAG.RemoveDeadNodes(); @@ -458,17 +306,7 @@ // If the first result of this node has been already legalized, then it cannot // reach N. - switch (getTypeAction(N->getValueType(0))) { - case Legal: - if (LegalizedNodes.count(SDValue(N, 0))) return false; - break; - case Promote: - if (PromotedNodes.count(SDValue(N, 0))) return false; - break; - case Expand: - if (ExpandedNodes.count(SDValue(N, 0))) return false; - break; - } + if (LegalizedNodes.count(SDValue(N, 0))) return false; // Okay, this node has not already been legalized. Check and legalize all // operands. If none lead to Dest, then we can legalize this node. @@ -483,59 +321,10 @@ } // Okay, this node looks safe, legalize it and return false. - HandleOp(SDValue(N, 0)); + LegalizeOp(SDValue(N, 0)); return false; } -/// HandleOp - Legalize, Promote, Widen, or Expand the specified operand as -/// appropriate for its type. -void SelectionDAGLegalize::HandleOp(SDValue Op) { - MVT VT = Op.getValueType(); - // If the type legalizer was run then we should never see any illegal result - // types here except for target constants (the type legalizer does not touch - // those) or for build vector used as a mask for a vector shuffle. - assert((TypesNeedLegalizing || getTypeAction(VT) == Legal || - IsLegalizingCallArgs || Op.getOpcode() == ISD::TargetConstant) && - "Illegal type introduced after type legalization?"); - switch (getTypeAction(VT)) { - default: assert(0 && "Bad type action!"); - case Legal: (void)LegalizeOp(Op); break; - case Promote: - if (!VT.isVector()) { - (void)PromoteOp(Op); - break; - } - else { - // See if we can widen otherwise use Expand to either scalarize or split - MVT WidenVT = TLI.getWidenVectorType(VT); - if (WidenVT != MVT::Other) { - (void) WidenVectorOp(Op, WidenVT); - break; - } - // else fall thru to expand since we can't widen the vector - } - case Expand: - if (!VT.isVector()) { - // If this is an illegal scalar, expand it into its two component - // pieces. - SDValue X, Y; - if (Op.getOpcode() == ISD::TargetConstant) - break; // Allow illegal target nodes. - ExpandOp(Op, X, Y); - } else if (VT.getVectorNumElements() == 1) { - // If this is an illegal single element vector, convert it to a - // scalar operation. - (void)ScalarizeVectorOp(Op); - } else { - // This is an illegal multiple element vector. - // Split it in half and legalize both parts. - SDValue X, Y; - SplitVectorOp(Op, X, Y); - } - break; - } -} - /// ExpandConstantFP - Expands the ConstantFP node to an integer constant or /// a load from the constant pool. static SDValue ExpandConstantFP(ConstantFPSDNode *CFP, bool UseCP, @@ -866,58 +655,6 @@ return DAG.getMergeValues(Ops, 2, dl); } -/// UnrollVectorOp - We know that the given vector has a legal type, however -/// the operation it performs is not legal and is an operation that we have -/// no way of lowering. "Unroll" the vector, splitting out the scalars and -/// operating on each element individually. -SDValue SelectionDAGLegalize::UnrollVectorOp(SDValue Op) { - MVT VT = Op.getValueType(); - assert(isTypeLegal(VT) && - "Caller should expand or promote operands that are not legal!"); - assert(Op.getNode()->getNumValues() == 1 && - "Can't unroll a vector with multiple results!"); - unsigned NE = VT.getVectorNumElements(); - MVT EltVT = VT.getVectorElementType(); - DebugLoc dl = Op.getDebugLoc(); - - SmallVector<SDValue, 8> Scalars; - SmallVector<SDValue, 4> Operands(Op.getNumOperands()); - for (unsigned i = 0; i != NE; ++i) { - for (unsigned j = 0; j != Op.getNumOperands(); ++j) { - SDValue Operand = Op.getOperand(j); - MVT OperandVT = Operand.getValueType(); - if (OperandVT.isVector()) { - // A vector operand; extract a single element. - MVT OperandEltVT = OperandVT.getVectorElementType(); - Operands[j] = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, - OperandEltVT, - Operand, - DAG.getConstant(i, MVT::i32)); - } else { - // A scalar operand; just use it as is. - Operands[j] = Operand; - } - } - - switch (Op.getOpcode()) { - default: - Scalars.push_back(DAG.getNode(Op.getOpcode(), dl, EltVT, - &Operands[0], Operands.size())); - break; - case ISD::SHL: - case ISD::SRA: - case ISD::SRL: - case ISD::ROTL: - case ISD::ROTR: - Scalars.push_back(DAG.getNode(Op.getOpcode(), dl, EltVT, Operands[0], - DAG.getShiftAmountOperand(Operands[1]))); - break; - } - } - - return DAG.getNode(ISD::BUILD_VECTOR, dl, VT, &Scalars[0], Scalars.size()); -} - /// GetFPLibCall - Return the right libcall for the given floating point type. static RTLIB::Libcall GetFPLibCall(MVT VT, RTLIB::Libcall Call_F32, @@ -985,21 +722,14 @@ if (Op.getOpcode() == ISD::TargetConstant) // Allow illegal target nodes. return Op; - assert(isTypeLegal(Op.getValueType()) && - "Caller should expand or promote operands that are not legal!"); SDNode *Node = Op.getNode(); DebugLoc dl = Node->getDebugLoc(); // If this operation defines any values that cannot be represented in a // register on this target, make sure to expand or promote them. - if (Node->getNumValues() > 1) { - for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i) - if (getTypeAction(Node->getValueType(i)) != Legal) { - HandleOp(Op.getValue(i)); - assert(LegalizedNodes.count(Op) && - "Handling didn't add legal operands!"); - return LegalizedNodes[Op]; - } + for (unsigned i = 0, e = Node->getNumValues(); i != e; ++i) + assert(getTypeAction(Node->getValueType(i)) == Legal && + "Unexpected illegal type!"); { } // Note that LegalizeOp may be reentered even from single-use nodes, which @@ -1298,20 +1028,13 @@ if (Result.getNode()) break; case TargetLowering::Legal: { - LegalizeAction Action = getTypeAction(Node->getOperand(1).getValueType()); - if (Action == Legal && Tmp1 == Node->getOperand(0)) + if (Tmp1 == Node->getOperand(0)) break; SmallVector<SDValue, 8> Ops; Ops.push_back(Tmp1); - if (Action == Legal) { - Ops.push_back(Node->getOperand(1)); // line # must be legal. - Ops.push_back(Node->getOperand(2)); // col # must be legal. - } else { - // Otherwise promote them. - Ops.push_back(PromoteOp(Node->getOperand(1))); - Ops.push_back(PromoteOp(Node->getOperand(2))); - } + Ops.push_back(Node->getOperand(1)); // line # must be legal. + Ops.push_back(Node->getOperand(2)); // col # must be legal. Ops.push_back(Node->getOperand(3)); // filename must be legal. Ops.push_back(Node->getOperand(4)); // working dir # must be legal. Result = DAG.UpdateNodeOperands(Result, &Ops[0], Ops.size()); @@ -1341,19 +1064,12 @@ switch (TLI.getOperationAction(ISD::DEBUG_LOC, MVT::Other)) { default: assert(0 && "This action is not supported yet!"); case TargetLowering::Legal: { - LegalizeAction Action = getTypeAction(Node->getOperand(1).getValueType()); Tmp1 = LegalizeOp(Node->getOperand(0)); // Legalize the chain. - if (Action == Legal && Tmp1 == Node->getOperand(0)) + if (Tmp1 == Node->getOperand(0)) break; - if (Action == Legal) { - Tmp2 = Node->getOperand(1); - Tmp3 = Node->getOperand(2); - Tmp4 = Node->getOperand(3); - } else { - Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the line #. - Tmp3 = LegalizeOp(Node->getOperand(2)); // Legalize the col #. - Tmp4 = LegalizeOp(Node->getOperand(3)); // Legalize the source file id. - } + Tmp2 = Node->getOperand(1); + Tmp3 = Node->getOperand(2); + Tmp4 = Node->getOperand(3); Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Tmp3, Tmp4); break; } @@ -1402,8 +1118,6 @@ Ops[0] = LegalizeOp(Node->getOperand(0)); // Legalize the chain. for (int x = 1; x < 6; ++x) { Ops[x] = Node->getOperand(x); - if (!isTypeLegal(Ops[x].getValueType())) - Ops[x] = PromoteOp(Ops[x]); } Result = DAG.UpdateNodeOperands(Result, &Ops[0], 6); break; @@ -1590,17 +1304,7 @@ // The type of the value to insert may not be legal, even though the vector // type is legal. Legalize/Promote accordingly. We do not handle Expand // here. - switch (getTypeAction(Node->getOperand(1).getValueType())) { - default: assert(0 && "Cannot expand insert element operand"); - case Legal: Tmp2 = LegalizeOp(Node->getOperand(1)); break; - case Promote: Tmp2 = PromoteOp(Node->getOperand(1)); break; - case Expand: - // FIXME: An alternative would be to check to see if the target is not - // going to custom lower this operation, we could bitcast to half elt - // width and perform two inserts at that width, if that is legal. - Tmp2 = Node->getOperand(1); - break; - } + Tmp2 = LegalizeOp(Node->getOperand(1)); Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Tmp3); switch (TLI.getOperationAction(ISD::INSERT_VECTOR_ELT, @@ -1650,11 +1354,6 @@ } break; case ISD::SCALAR_TO_VECTOR: - if (!TLI.isTypeLegal(Node->getOperand(0).getValueType())) { - Result = LegalizeOp(ExpandSCALAR_TO_VECTOR(Node)); - break; - } - Tmp1 = LegalizeOp(Node->getOperand(0)); // InVal Result = DAG.UpdateNodeOperands(Result, Tmp1); switch (TLI.getOperationAction(ISD::SCALAR_TO_VECTOR, @@ -1746,7 +1445,9 @@ Tmp1 = Node->getOperand(0); Tmp2 = LegalizeOp(Node->getOperand(1)); Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2); - Result = ExpandEXTRACT_SUBVECTOR(Result); + assert(TLI.isOperationLegal(ISD::EXTRACT_SUBVECTOR, + Node->getValueType(0)) && + "Don't know how to legalize extract_subvector!"); break; case ISD::CONCAT_VECTORS: { @@ -1798,13 +1499,11 @@ // Recursively Legalize all of the inputs of the call end that do not lead // to this call start. This ensures that any libcalls that need be inserted // are inserted *before* the CALLSEQ_START. - IsLegalizingCallArgs = true; {SmallPtrSet<SDNode*, 32> NodesLeadingTo; for (unsigned i = 0, e = CallEnd->getNumOperands(); i != e; ++i) LegalizeAllNodesNotLeadingTo(CallEnd->getOperand(i).getNode(), Node, NodesLeadingTo); } - IsLegalizingCallArgs = false; // Now that we legalized all of the inputs (which may have inserted // libcalls) create the new CALLSEQ_START node. @@ -1994,12 +1693,7 @@ Tmp1 = LegalizeOp(Tmp1); LastCALLSEQ_END = DAG.getEntryNode(); - switch (getTypeAction(Node->getOperand(1).getValueType())) { - default: assert(0 && "Indirect target must be legal type (pointer)!"); - case Legal: - Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the condition. - break; - } + Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the condition. Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2); break; case ISD::BR_JT: @@ -2053,24 +1747,8 @@ Tmp1 = LegalizeOp(Tmp1); LastCALLSEQ_END = DAG.getEntryNode(); - switch (getTypeAction(Node->getOperand(1).getValueType())) { - case Expand: assert(0 && "It's impossible to expand bools"); - case Legal: - Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the condition. - break; - case Promote: { - Tmp2 = PromoteOp(Node->getOperand(1)); // Promote the condition. + Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the condition. - // The top bits of the promoted condition are not necessarily zero, ensure - // that the value is properly zero extended. - unsigned BitWidth = Tmp2.getValueSizeInBits(); - if (!DAG.MaskedValueIsZero(Tmp2, - APInt::getHighBitsSet(BitWidth, BitWidth-1))) - Tmp2 = DAG.getZeroExtendInReg(Tmp2, dl, MVT::i1); - break; - } - } - // Basic block destination (Op#2) is always legal. Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Node->getOperand(2)); @@ -2386,29 +2064,16 @@ } case ISD::EXTRACT_ELEMENT: { MVT OpTy = Node->getOperand(0).getValueType(); - switch (getTypeAction(OpTy)) { - default: assert(0 && "EXTRACT_ELEMENT action for type unimplemented!"); - case Legal: - if (cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue()) { - // 1 -> Hi - Result = DAG.getNode(ISD::SRL, dl, OpTy, Node->getOperand(0), - DAG.getConstant(OpTy.getSizeInBits()/2, - TLI.getShiftAmountTy())); - Result = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), Result); - } else { - // 0 -> Lo - Result = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), - Node->getOperand(0)); - } - break; - case Expand: - // Get both the low and high parts. - ExpandOp(Node->getOperand(0), Tmp1, Tmp2); - if (cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue()) - Result = Tmp2; // 1 -> Hi - else - Result = Tmp1; // 0 -> Lo - break; + if (cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue()) { + // 1 -> Hi + Result = DAG.getNode(ISD::SRL, dl, OpTy, Node->getOperand(0), + DAG.getConstant(OpTy.getSizeInBits()/2, + TLI.getShiftAmountTy())); + Result = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), Result); + } else { + // 0 -> Lo + Result = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), + Node->getOperand(0)); } break; } @@ -2452,67 +2117,7 @@ case 3: // ret val Tmp2 = Node->getOperand(1); Tmp3 = Node->getOperand(2); // Signness - switch (getTypeAction(Tmp2.getValueType())) { - case Legal: - Result = DAG.UpdateNodeOperands(Result, Tmp1, LegalizeOp(Tmp2), Tmp3); - break; - case Expand: - if (!Tmp2.getValueType().isVector()) { - SDValue Lo, Hi; - ExpandOp(Tmp2, Lo, Hi); - - // Big endian systems want the hi reg first. - if (TLI.isBigEndian()) - std::swap(Lo, Hi); - - if (Hi.getNode()) - Result = DAG.getNode(ISD::RET, dl, MVT::Other, - Tmp1, Lo, Tmp3, Hi, Tmp3); - else - Result = DAG.getNode(ISD::RET, dl, MVT::Other, Tmp1, Lo, Tmp3); - Result = LegalizeOp(Result); - } else { - SDNode *InVal = Tmp2.getNode(); - int InIx = Tmp2.getResNo(); - unsigned NumElems = InVal->getValueType(InIx).getVectorNumElements(); - MVT EVT = InVal->getValueType(InIx).getVectorElementType(); - - // Figure out if there is a simple type corresponding to this Vector - // type. If so, convert to the vector type. - MVT TVT = MVT::getVectorVT(EVT, NumElems); - if (TLI.isTypeLegal(TVT)) { - // Turn this into a return of the vector type. - Tmp2 = LegalizeOp(Tmp2); - Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Tmp3); - } else if (NumElems == 1) { - // Turn this into a return of the scalar type. - Tmp2 = ScalarizeVectorOp(Tmp2); - Tmp2 = LegalizeOp(Tmp2); - Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Tmp3); - - // FIXME: Returns of gcc generic vectors smaller than a legal type - // should be returned in integer registers! - - // The scalarized value type may not be legal, e.g. it might require - // promotion or expansion. Relegalize the return. - Result = LegalizeOp(Result); - } else { - // FIXME: Returns of gcc generic vectors larger than a legal vector - // type should be returned by reference! - SDValue Lo, Hi; - SplitVectorOp(Tmp2, Lo, Hi); - Result = DAG.getNode(ISD::RET, dl, MVT::Other, - Tmp1, Lo, Tmp3, Hi, Tmp3); - Result = LegalizeOp(Result); - } - } - break; - case Promote: - Tmp2 = PromoteOp(Node->getOperand(1)); - Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, Tmp3); - Result = LegalizeOp(Result); - break; - } + Result = DAG.UpdateNodeOperands(Result, Tmp1, LegalizeOp(Tmp2), Tmp3); break; case 1: // ret void Result = DAG.UpdateNodeOperands(Result, Tmp1); @@ -2520,34 +2125,12 @@ default: { // ret <values> SmallVector<SDValue, 8> NewValues; NewValues.push_back(Tmp1); - for (unsigned i = 1, e = Node->getNumOperands(); i < e; i += 2) - switch (getTypeAction(Node->getOperand(i).getValueType())) { - case Legal: - NewValues.push_back(LegalizeOp(Node->getOperand(i))); - NewValues.push_back(Node->getOperand(i+1)); - break; - case Expand: { - SDValue Lo, Hi; - assert(!Node->getOperand(i).getValueType().isExtended() && - "FIXME: TODO: implement returning non-legal vector types!"); - ExpandOp(Node->getOperand(i), Lo, Hi); - NewValues.push_back(Lo); - NewValues.push_back(Node->getOperand(i+1)); - if (Hi.getNode()) { - NewValues.push_back(Hi); - NewValues.push_back(Node->getOperand(i+1)); - } - break; - } - case Promote: - assert(0 && "Can't promote multiple return value yet!"); - } + for (unsigned i = 1, e = Node->getNumOperands(); i < e; i += 2) { + NewValues.push_back(LegalizeOp(Node->getOperand(i))); + NewValues.push_back(Node->getOperand(i+1)); + } - if (NewValues.size() == Node->getNumOperands()) - Result = DAG.UpdateNodeOperands(Result, &NewValues[0],NewValues.size()); - else - Result = DAG.getNode(ISD::RET, dl, MVT::Other, - &NewValues[0], NewValues.size()); + Result = DAG.UpdateNodeOperands(Result, &NewValues[0],NewValues.size()); break; } } @@ -2618,8 +2201,7 @@ } } - switch (getTypeAction(ST->getMemoryVT())) { - case Legal: { + { Tmp3 = LegalizeOp(ST->getValue()); Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp3, Tmp2, ST->getOffset()); @@ -2653,111 +2235,8 @@ } break; } - case Promote: - if (!ST->getMemoryVT().isVector()) { - // Truncate the value and store the result. - Tmp3 = PromoteOp(ST->getValue()); - Result = DAG.getTruncStore(Tmp1, dl, Tmp3, Tmp2, ST->getSrcValue(), - SVOffset, ST->getMemoryVT(), - isVolatile, Alignment); - break; - } - // Fall thru to expand for vector - case Expand: { - unsigned IncrementSize = 0; - SDValue Lo, Hi; - - // If this is a vector type, then we have to calculate the increment as - // the product of the element size in bytes, and the number of elements - // in the high half of the vector. - if (ST->getValue().getValueType().isVector()) { - SDNode *InVal = ST->getValue().getNode(); - int InIx = ST->getValue().getResNo(); - MVT InVT = InVal->getValueType(InIx); - unsigned NumElems = InVT.getVectorNumElements(); - MVT EVT = InVT.getVectorElementType(); - - // Figure out if there is a simple type corresponding to this Vector - // type. If so, convert to the vector type. - MVT TVT = MVT::getVectorVT(EVT, NumElems); - if (TLI.isTypeLegal(TVT)) { - // Turn this into a normal store of the vector type. - Tmp3 = LegalizeOp(ST->getValue()); - Result = DAG.getStore(Tmp1, dl, Tmp3, Tmp2, ST->getSrcValue(), - SVOffset, isVolatile, Alignment); - Result = LegalizeOp(Result); - break; - } else if (NumElems == 1) { - // Turn this into a normal store of the scalar type. - Tmp3 = ScalarizeVectorOp(ST->getValue()); - Result = DAG.getStore(Tmp1, dl, Tmp3, Tmp2, ST->getSrcValue(), - SVOffset, isVolatile, Alignment); - // The scalarized value type may not be legal, e.g. it might require - // promotion or expansion. Relegalize the scalar store. - Result = LegalizeOp(Result); - break; - } else { - // Check if we have widen this node with another value - std::map<SDValue, SDValue>::iterator I - WidenNodes.find(ST->getValue()); - if (I != WidenNodes.end()) { - Result = StoreWidenVectorOp(ST, Tmp1, Tmp2); - break; - } - else { - SplitVectorOp(ST->getValue(), Lo, Hi); - IncrementSize = Lo.getNode()->getValueType(0).getVectorNumElements() * - EVT.getSizeInBits()/8; - } - } - } else { - ExpandOp(ST->getValue(), Lo, Hi); - IncrementSize = Hi.getNode() ? Hi.getValueType().getSizeInBits()/8 : 0; - - if (Hi.getNode() && TLI.isBigEndian()) - std::swap(Lo, Hi); - } - - Lo = DAG.getStore(Tmp1, dl, Lo, Tmp2, ST->getSrcValue(), - SVOffset, isVolatile, Alignment); - - if (Hi.getNode() == NULL) { - // Must be int <-> float one-to-one expansion. - Result = Lo; - break; - } - - Tmp2 = DAG.getNode(ISD::ADD, dl, Tmp2.getValueType(), Tmp2, - DAG.getIntPtrConstant(IncrementSize)); - assert(isTypeLegal(Tmp2.getValueType()) && - "Pointers must be legal!"); - SVOffset += IncrementSize; - Alignment = MinAlign(Alignment, IncrementSize); - Hi = DAG.getStore(Tmp1, dl, Hi, Tmp2, ST->getSrcValue(), - SVOffset, isVolatile, Alignment); - Result = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo, Hi); - break; - } // case Expand - } } else { - switch (getTypeAction(ST->getValue().getValueType())) { - case Legal: - Tmp3 = LegalizeOp(ST->getValue()); - break; - case Promote: - if (!ST->getValue().getValueType().isVector()) { - // We can promote the value, the truncstore will still take care of it. - Tmp3 = PromoteOp(ST->getValue()); - break; - } - // Vector case falls through to expand - case Expand: - // Just store the low part. This may become a non-trunc store, so make - // sure to use getTruncStore, not UpdateNodeOperands below. - ExpandOp(ST->getValue(), Tmp3, Tmp4); - return DAG.getTruncStore(Tmp1, dl, Tmp3, Tmp2, ST->getSrcValue(), - SVOffset, MVT::i8, isVolatile, Alignment); - } + Tmp3 = LegalizeOp(ST->getValue()); MVT StVT = ST->getMemoryVT(); unsigned StWidth = StVT.getSizeInBits(); @@ -2943,22 +2422,7 @@ return Result; case ISD::SELECT: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "It's impossible to expand bools"); - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); // Legalize the condition. - break; - case Promote: { - assert(!Node->getOperand(0).getValueType().isVector() && "not possible"); - Tmp1 = PromoteOp(Node->getOperand(0)); // Promote the condition. - // Make sure the condition is either zero or one. - unsigned BitWidth = Tmp1.getValueSizeInBits(); - if (!DAG.MaskedValueIsZero(Tmp1, - APInt::getHighBitsSet(BitWidth, BitWidth-1))) - Tmp1 = DAG.getZeroExtendInReg(Tmp1, dl, MVT::i1); - break; - } - } + Tmp1 = LegalizeOp(Node->getOperand(0)); // Legalize the condition. Tmp2 = LegalizeOp(Node->getOperand(1)); // TrueVal Tmp3 = LegalizeOp(Node->getOperand(2)); // FalseVal @@ -3115,39 +2579,9 @@ SDValue CC = Node->getOperand(2); Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2, CC); - - // Everything is legal, see if we should expand this op or something. - switch (TLI.getOperationAction(ISD::VSETCC, Tmp1.getValueType())) { - default: assert(0 && "This action is not supported yet!"); - case TargetLowering::Legal: break; - case TargetLowering::Custom: - Tmp1 = TLI.LowerOperation(Result, DAG); - if (Tmp1.getNode()) Result = Tmp1; - break; - case TargetLowering::Expand: { - // Unroll into a nasty set of scalar code for now. - MVT VT = Node->getValueType(0); - unsigned NumElems = VT.getVectorNumElements(); - MVT EltVT = VT.getVectorElementType(); - MVT TmpEltVT = Tmp1.getValueType().getVectorElementType(); - SmallVector<SDValue, 8> Ops(NumElems); - for (unsigned i = 0; i < NumElems; ++i) { - SDValue In1 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, - Tmp1, DAG.getIntPtrConstant(i)); - Ops[i] = DAG.getNode(ISD::SETCC, dl, TLI.getSetCCResultType(TmpEltVT), - In1, DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, - TmpEltVT, Tmp2, - DAG.getIntPtrConstant(i)), - CC); - Ops[i] = DAG.getNode(ISD::SELECT, dl, EltVT, Ops[i], - DAG.getConstant(APInt::getAllOnesValue - (EltVT.getSizeInBits()), EltVT), - DAG.getConstant(0, EltVT)); - } - Result = DAG.getNode(ISD::BUILD_VECTOR, dl, VT, &Ops[0], NumElems); - break; - } - } + + assert(TLI.isOperationLegal(ISD::VSETCC, Tmp1.getValueType()) && + "Illegal vector ops should be unrolled earlier!"); break; } @@ -3221,15 +2655,7 @@ !Node->getValueType(0).isVector()) Tmp2 = DAG.getShiftAmountOperand(Tmp2); - switch (getTypeAction(Tmp2.getValueType())) { - case Expand: assert(0 && "Not possible"); - case Legal: - Tmp2 = LegalizeOp(Tmp2); // Legalize the RHS. - break; - case Promote: - Tmp2 = PromoteOp(Tmp2); // Promote the RHS. - break; - } + Tmp2 = LegalizeOp(Tmp2); // Legalize the RHS. Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2); @@ -3348,10 +2774,7 @@ break; } - assert(Node->getValueType(0).isVector() && - "Cannot expand this binary operator!"); - // Expand the operation into a bunch of nasty scalar code. - Result = LegalizeOp(UnrollVectorOp(Op)); + assert(0 && "Cannot expand this binary operator!"); break; } case TargetLowering::Promote: { @@ -3392,15 +2815,7 @@ case ISD::FCOPYSIGN: // FCOPYSIGN does not require LHS/RHS to match type! Tmp1 = LegalizeOp(Node->getOperand(0)); // LHS - switch (getTypeAction(Node->getOperand(1).getValueType())) { - case Expand: assert(0 && "Not possible"); - case Legal: - Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the RHS. - break; - case Promote: - Tmp2 = PromoteOp(Node->getOperand(1)); // Promote the RHS. - break; - } + Tmp2 = LegalizeOp(Node->getOperand(1)); // Legalize the RHS. Result = DAG.UpdateNodeOperands(Result, Tmp1, Tmp2); @@ -3503,7 +2918,6 @@ case ISD::BUILD_PAIR: { MVT PairTy = Node->getValueType(0); - // TODO: handle the case where the Lo and Hi operands are not of legal type Tmp1 = LegalizeOp(Node->getOperand(0)); // Lo Tmp2 = LegalizeOp(Node->getOperand(1)); // Hi switch (TLI.getOperationAction(ISD::BUILD_PAIR, PairTy)) { @@ -3601,10 +3015,7 @@ break; } - assert(VT.isVector() && - "Cannot expand this binary operator!"); - // Expand the operation into a bunch of nasty scalar code. - Result = LegalizeOp(UnrollVectorOp(Op)); + assert(0 && "Cannot expand this binary operator!"); break; } } @@ -3885,11 +3296,7 @@ case ISD::FNEARBYINT: { MVT VT = Node->getValueType(0); - // Expand unsupported unary vector operators by unrolling them. - if (VT.isVector()) { - Result = LegalizeOp(UnrollVectorOp(Op)); - break; - } + assert(!VT.isVector() && "Vector should be already expanded!"); RTLIB::Libcall LC = RTLIB::UNKNOWN_LIBCALL; switch(Node->getOpcode()) { @@ -3959,11 +3366,7 @@ case ISD::FPOWI: { MVT VT = Node->getValueType(0); - // Expand unsupported unary vector operators by unrolling them. - if (VT.isVector()) { - Result = LegalizeOp(UnrollVectorOp(Op)); - break; - } + assert(!VT.isVector() && "Vector should be already expanded!"); // We always lower FPOWI into a libcall. No target support for it yet. RTLIB::Libcall LC = GetFPLibCall(VT, RTLIB::POWI_F32, RTLIB::POWI_F64, @@ -3973,47 +3376,17 @@ break; } case ISD::BIT_CONVERT: - if (!isTypeLegal(Node->getOperand(0).getValueType())) { + switch (TLI.getOperationAction(ISD::BIT_CONVERT, + Node->getOperand(0).getValueType())) { + default: assert(0 && "Unknown operation action!"); + case TargetLowering::Expand: Result = EmitStackConvert(Node->getOperand(0), Node->getValueType(0), Node->getValueType(0), dl); - } else if (Op.getOperand(0).getValueType().isVector()) { - // The input has to be a vector type, we have to either scalarize it, pack - // it, or convert it based on whether the input vector type is legal. - SDNode *InVal = Node->getOperand(0).getNode(); - int InIx = Node->getOperand(0).getResNo(); - unsigned NumElems = InVal->getValueType(InIx).getVectorNumElements(); - MVT EVT = InVal->getValueType(InIx).getVectorElementType(); - - // Figure out if there is a simple type corresponding to this Vector - // type. If so, convert to the vector type. - MVT TVT = MVT::getVectorVT(EVT, NumElems); - if (TLI.isTypeLegal(TVT)) { - // Turn this into a bit convert of the vector input. - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::BIT_CONVERT, dl, Node->getValueType(0), Tmp1); - break; - } else if (NumElems == 1) { - // Turn this into a bit convert of the scalar input. - Result = DAG.getNode(ISD::BIT_CONVERT, dl, Node->getValueType(0), - ScalarizeVectorOp(Node->getOperand(0))); - break; - } else { - // FIXME: UNIMP! Store then reload - assert(0 && "Cast from unsupported vector type not implemented yet!"); - } - } else { - switch (TLI.getOperationAction(ISD::BIT_CONVERT, - Node->getOperand(0).getValueType())) { - default: assert(0 && "Unknown operation action!"); - case TargetLowering::Expand: - Result = EmitStackConvert(Node->getOperand(0), Node->getValueType(0), - Node->getValueType(0), dl); - break; - case TargetLowering::Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, Tmp1); - break; - } + break; + case TargetLowering::Legal: + Tmp1 = LegalizeOp(Node->getOperand(0)); + Result = DAG.UpdateNodeOperands(Result, Tmp1); + break; } break; case ISD::CONVERT_RNDSAT: { @@ -4034,25 +3407,14 @@ SDValue STyOp = Node->getOperand(2); SDValue RndOp = Node->getOperand(3); SDValue SatOp = Node->getOperand(4); - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "Shouldn't need to expand other operators here!"); - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, Tmp1, DTyOp, STyOp, - RndOp, SatOp); - if (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0)) =- TargetLowering::Custom) { - Tmp1 = TLI.LowerOperation(Result, DAG); - if (Tmp1.getNode()) Result = Tmp1; - } - break; - case Promote: - Result = PromoteOp(Node->getOperand(0)); - // For FP, make Op1 a i32 - - Result = DAG.getConvertRndSat(Op.getValueType(), dl, Result, - DTyOp, STyOp, RndOp, SatOp, CvtCode); - break; + + Tmp1 = LegalizeOp(Node->getOperand(0)); + Result = DAG.UpdateNodeOperands(Result, Tmp1, DTyOp, STyOp, + RndOp, SatOp); + if (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0)) =+ TargetLowering::Custom) { + Tmp1 = TLI.LowerOperation(Result, DAG); + if (Tmp1.getNode()) Result = Tmp1; } break; } @@ -4068,45 +3430,25 @@ break; } case ISD::TRUNCATE: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - switch (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0))) { - default: assert(0 && "Unknown TRUNCATE legalization operation action!"); - case TargetLowering::Custom: - isCustom = true; - // FALLTHROUGH - case TargetLowering::Legal: - Result = DAG.UpdateNodeOperands(Result, Tmp1); - if (isCustom) { - Tmp1 = TLI.LowerOperation(Result, DAG); - if (Tmp1.getNode()) Result = Tmp1; - } - break; - case TargetLowering::Expand: - assert(Result.getValueType().isVector() && "must be vector type"); - // Unroll the truncate. We should do better. - Result = LegalizeOp(UnrollVectorOp(Result)); + Tmp1 = LegalizeOp(Node->getOperand(0)); + switch (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0))) { + default: assert(0 && "Unknown TRUNCATE legalization operation action!"); + case TargetLowering::Custom: + isCustom = true; + // FALLTHROUGH + case TargetLowering::Legal: + Result = DAG.UpdateNodeOperands(Result, Tmp1); + if (isCustom) { + Tmp1 = TLI.LowerOperation(Result, DAG); + if (Tmp1.getNode()) Result = Tmp1; } break; - case Expand: - ExpandOp(Node->getOperand(0), Tmp1, Tmp2); - - // Since the result is legal, we should just be able to truncate the low - // part of the source. - Result = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), Tmp1); - break; - case Promote: - Result = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::TRUNCATE, dl, Op.getValueType(), Result); - break; } break; case ISD::FP_TO_SINT: case ISD::FP_TO_UINT: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Legal: + { Tmp1 = LegalizeOp(Node->getOperand(0)); switch (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0))){ @@ -4152,51 +3494,7 @@ } break; } - break; - case Expand: { - MVT VT = Op.getValueType(); - MVT OVT = Node->getOperand(0).getValueType(); - // Convert ppcf128 to i32 - if (OVT == MVT::ppcf128 && VT == MVT::i32) { - if (Node->getOpcode() == ISD::FP_TO_SINT) { - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, MVT::ppcf128, - Node->getOperand(0), DAG.getValueType(MVT::f64)); - Result = DAG.getNode(ISD::FP_ROUND, dl, MVT::f64, Result, - DAG.getIntPtrConstant(1)); - Result = DAG.getNode(ISD::FP_TO_SINT, dl, VT, Result); - } else { - const uint64_t TwoE31[] = {0x41e0000000000000LL, 0}; - APFloat apf = APFloat(APInt(128, 2, TwoE31)); - Tmp2 = DAG.getConstantFP(apf, OVT); - // X>=2^31 ? (int)(X-2^31)+0x80000000 : (int)X - // FIXME: generated code sucks. - Result = DAG.getNode(ISD::SELECT_CC, dl, VT, Node->getOperand(0), - Tmp2, - DAG.getNode(ISD::ADD, dl, MVT::i32, - DAG.getNode(ISD::FP_TO_SINT, dl, VT, - DAG.getNode(ISD::FSUB, dl, OVT, - Node->getOperand(0), Tmp2)), - DAG.getConstant(0x80000000, MVT::i32)), - DAG.getNode(ISD::FP_TO_SINT, dl, VT, - Node->getOperand(0)), - DAG.getCondCode(ISD::SETGE)); - } - break; - } - // Convert f32 / f64 to i32 / i64 / i128. - RTLIB::Libcall LC = (Node->getOpcode() == ISD::FP_TO_SINT) ? - RTLIB::getFPTOSINT(OVT, VT) : RTLIB::getFPTOUINT(OVT, VT); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unexpectd fp-to-int conversion!"); - SDValue Dummy; - Result = ExpandLibCall(LC, Node, false/*sign irrelevant*/, Dummy); - break; } - case Promote: - Tmp1 = PromoteOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, LegalizeOp(Tmp1)); - Result = LegalizeOp(Result); - break; - } break; case ISD::FP_EXTEND: { @@ -4208,17 +3506,8 @@ Result = EmitStackConvert(Node->getOperand(0), SrcVT, DstVT, dl); break; } - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "Shouldn't need to expand other operators here!"); - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, Tmp1); - break; - case Promote: - Tmp1 = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::FP_EXTEND, dl, Op.getValueType(), Tmp1); - break; - } + Tmp1 = LegalizeOp(Node->getOperand(0)); + Result = DAG.UpdateNodeOperands(Result, Tmp1); break; } case ISD::FP_ROUND: { @@ -4226,67 +3515,27 @@ MVT SrcVT = Op.getOperand(0).getValueType(); if (TLI.getConvertAction(SrcVT, DstVT) == TargetLowering::Expand) { if (SrcVT == MVT::ppcf128) { - SDValue Lo; - ExpandOp(Node->getOperand(0), Lo, Result); - // Round it the rest of the way (e.g. to f32) if needed. - if (DstVT!=MVT::f64) - Result = DAG.getNode(ISD::FP_ROUND, dl, - DstVT, Result, Op.getOperand(1)); - break; + // FIXME: Figure out how to extract the double without + // help from type legalization } // The only other way we can lower this is to turn it into a STORE, // LOAD pair, targetting a temporary location (a stack slot). Result = EmitStackConvert(Node->getOperand(0), DstVT, DstVT, dl); break; } - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "Shouldn't need to expand other operators here!"); - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, Tmp1, Node->getOperand(1)); - break; - case Promote: - Tmp1 = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::FP_ROUND, dl, Op.getValueType(), Tmp1, - Node->getOperand(1)); - break; - } + Tmp1 = LegalizeOp(Node->getOperand(0)); + Result = DAG.UpdateNodeOperands(Result, Tmp1, Node->getOperand(1)); break; } case ISD::ANY_EXTEND: case ISD::ZERO_EXTEND: case ISD::SIGN_EXTEND: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "Shouldn't need to expand other operators here!"); - case Legal: - Tmp1 = LegalizeOp(Node->getOperand(0)); - Result = DAG.UpdateNodeOperands(Result, Tmp1); - if (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0)) =- TargetLowering::Custom) { - Tmp1 = TLI.LowerOperation(Result, DAG); - if (Tmp1.getNode()) Result = Tmp1; - } - break; - case Promote: - switch (Node->getOpcode()) { - case ISD::ANY_EXTEND: - Tmp1 = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::ANY_EXTEND, dl, Op.getValueType(), Tmp1); - break; - case ISD::ZERO_EXTEND: - Result = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::ANY_EXTEND, dl, Op.getValueType(), Result); - Result = DAG.getZeroExtendInReg(Result, dl, - Node->getOperand(0).getValueType()); - break; - case ISD::SIGN_EXTEND: - Result = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::ANY_EXTEND, dl, Op.getValueType(), Result); - Result = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, Result.getValueType(), - Result, - DAG.getValueType(Node->getOperand(0).getValueType())); - break; - } + Tmp1 = LegalizeOp(Node->getOperand(0)); + Result = DAG.UpdateNodeOperands(Result, Tmp1); + if (TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0)) =+ TargetLowering::Custom) { + Tmp1 = TLI.LowerOperation(Result, DAG); + if (Tmp1.getNode()) Result = Tmp1; } break; case ISD::FP_ROUND_INREG: @@ -4511,496 +3760,6 @@ return Result; } -/// PromoteOp - Given an operation that produces a value in an invalid type, -/// promote it to compute the value into a larger type. The produced value will -/// have the correct bits for the low portion of the register, but no guarantee -/// is made about the top bits: it may be zero, sign-extended, or garbage. -SDValue SelectionDAGLegalize::PromoteOp(SDValue Op) { - MVT VT = Op.getValueType(); - MVT NVT = TLI.getTypeToTransformTo(VT); - assert(getTypeAction(VT) == Promote && - "Caller should expand or legalize operands that are not promotable!"); - assert(NVT.bitsGT(VT) && NVT.isInteger() == VT.isInteger() && - "Cannot promote to smaller type!"); - - SDValue Tmp1, Tmp2, Tmp3; - SDValue Result; - SDNode *Node = Op.getNode(); - DebugLoc dl = Node->getDebugLoc(); - - DenseMap<SDValue, SDValue>::iterator I = PromotedNodes.find(Op); - if (I != PromotedNodes.end()) return I->second; - - switch (Node->getOpcode()) { - case ISD::CopyFromReg: - assert(0 && "CopyFromReg must be legal!"); - default: -#ifndef NDEBUG - cerr << "NODE: "; Node->dump(&DAG); cerr << "\n"; -#endif - assert(0 && "Do not know how to promote this operator!"); - abort(); - case ISD::UNDEF: - Result = DAG.getUNDEF(NVT); - break; - case ISD::Constant: - if (VT != MVT::i1) - Result = DAG.getNode(ISD::SIGN_EXTEND, dl, NVT, Op); - else - Result = DAG.getNode(ISD::ZERO_EXTEND, dl, NVT, Op); - assert(isa<ConstantSDNode>(Result) && "Didn't constant fold zext?"); - break; - case ISD::ConstantFP: - Result = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Op); - assert(isa<ConstantFPSDNode>(Result) && "Didn't constant fold fp_extend?"); - break; - - case ISD::SETCC: { - MVT VT0 = Node->getOperand(0).getValueType(); - assert(isTypeLegal(TLI.getSetCCResultType(VT0)) - && "SetCC type is not legal??"); - Result = DAG.getNode(ISD::SETCC, dl, TLI.getSetCCResultType(VT0), - Node->getOperand(0), Node->getOperand(1), - Node->getOperand(2)); - break; - } - case ISD::TRUNCATE: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Legal: - Result = LegalizeOp(Node->getOperand(0)); - assert(Result.getValueType().bitsGE(NVT) && - "This truncation doesn't make sense!"); - if (Result.getValueType().bitsGT(NVT)) // Truncate to NVT instead of VT - Result = DAG.getNode(ISD::TRUNCATE, dl, NVT, Result); - break; - case Promote: - // The truncation is not required, because we don't guarantee anything - // about high bits anyway. - Result = PromoteOp(Node->getOperand(0)); - break; - case Expand: - ExpandOp(Node->getOperand(0), Tmp1, Tmp2); - // Truncate the low part of the expanded value to the result type - Result = DAG.getNode(ISD::TRUNCATE, dl, NVT, Tmp1); - } - break; - case ISD::SIGN_EXTEND: - case ISD::ZERO_EXTEND: - case ISD::ANY_EXTEND: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "BUG: Smaller reg should have been promoted!"); - case Legal: - // Input is legal? Just do extend all the way to the larger type. - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Node->getOperand(0)); - break; - case Promote: - // Promote the reg if it's smaller. - Result = PromoteOp(Node->getOperand(0)); - // The high bits are not guaranteed to be anything. Insert an extend. - if (Node->getOpcode() == ISD::SIGN_EXTEND) - Result = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Result, - DAG.getValueType(Node->getOperand(0).getValueType())); - else if (Node->getOpcode() == ISD::ZERO_EXTEND) - Result = DAG.getZeroExtendInReg(Result, dl, - Node->getOperand(0).getValueType()); - break; - } - break; - case ISD::CONVERT_RNDSAT: { - ISD::CvtCode CvtCode = cast<CvtRndSatSDNode>(Node)->getCvtCode(); - assert ((CvtCode == ISD::CVT_SS || CvtCode == ISD::CVT_SU || - CvtCode == ISD::CVT_US || CvtCode == ISD::CVT_UU || - CvtCode == ISD::CVT_SF || CvtCode == ISD::CVT_UF) && - "can only promote integers"); - Result = DAG.getConvertRndSat(NVT, dl, Node->getOperand(0), - Node->getOperand(1), Node->getOperand(2), - Node->getOperand(3), Node->getOperand(4), - CvtCode); - break; - - } - case ISD::BIT_CONVERT: - Result = EmitStackConvert(Node->getOperand(0), Node->getValueType(0), - Node->getValueType(0), dl); - Result = PromoteOp(Result); - break; - - case ISD::FP_EXTEND: - assert(0 && "Case not implemented. Dynamically dead with 2 FP types!"); - case ISD::FP_ROUND: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "BUG: Cannot expand FP regs!"); - case Promote: assert(0 && "Unreachable with 2 FP types!"); - case Legal: - if (Node->getConstantOperandVal(1) == 0) { - // Input is legal? Do an FP_ROUND_INREG. - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Node->getOperand(0), - DAG.getValueType(VT)); - } else { - // Just remove the truncate, it isn't affecting the value. - Result = DAG.getNode(ISD::FP_ROUND, dl, NVT, Node->getOperand(0), - Node->getOperand(1)); - } - break; - } - break; - case ISD::SINT_TO_FP: - case ISD::UINT_TO_FP: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Legal: - // No extra round required here. - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Node->getOperand(0)); - break; - - case Promote: - Result = PromoteOp(Node->getOperand(0)); - if (Node->getOpcode() == ISD::SINT_TO_FP) - Result = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, Result.getValueType(), - Result, - DAG.getValueType(Node->getOperand(0).getValueType())); - else - Result = DAG.getZeroExtendInReg(Result, dl, - Node->getOperand(0).getValueType()); - // No extra round required here. - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Result); - break; - case Expand: - Result = ExpandIntToFP(Node->getOpcode() == ISD::SINT_TO_FP, NVT, - Node->getOperand(0), dl); - // Round if we cannot tolerate excess precision. - if (NoExcessFPPrecision) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - } - break; - - case ISD::SIGN_EXTEND_INREG: - Result = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Result, - Node->getOperand(1)); - break; - case ISD::FP_TO_SINT: - case ISD::FP_TO_UINT: - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Legal: - case Expand: - Tmp1 = Node->getOperand(0); - break; - case Promote: - // The input result is prerounded, so we don't have to do anything - // special. - Tmp1 = PromoteOp(Node->getOperand(0)); - break; - } - // If we're promoting a UINT to a larger size, check to see if the new node - // will be legal. If it isn't, check to see if FP_TO_SINT is legal, since - // we can use that instead. This allows us to generate better code for - // FP_TO_UINT for small destination sizes on targets where FP_TO_UINT is not - // legal, such as PowerPC. - if (Node->getOpcode() == ISD::FP_TO_UINT && - !TLI.isOperationLegalOrCustom(ISD::FP_TO_UINT, NVT) && - (TLI.isOperationLegalOrCustom(ISD::FP_TO_SINT, NVT) || - TLI.getOperationAction(ISD::FP_TO_SINT, NVT)==TargetLowering::Custom)){ - Result = DAG.getNode(ISD::FP_TO_SINT, dl, NVT, Tmp1); - } else { - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1); - } - break; - - case ISD::FABS: - case ISD::FNEG: - Tmp1 = PromoteOp(Node->getOperand(0)); - assert(Tmp1.getValueType() == NVT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1); - // NOTE: we do not have to do any extra rounding here for - // NoExcessFPPrecision, because we know the input will have the appropriate - // precision, and these operations don't modify precision at all. - break; - - case ISD::FLOG: - case ISD::FLOG2: - case ISD::FLOG10: - case ISD::FEXP: - case ISD::FEXP2: - case ISD::FSQRT: - case ISD::FSIN: - case ISD::FCOS: - case ISD::FTRUNC: - case ISD::FFLOOR: - case ISD::FCEIL: - case ISD::FRINT: - case ISD::FNEARBYINT: - Tmp1 = PromoteOp(Node->getOperand(0)); - assert(Tmp1.getValueType() == NVT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1); - if (NoExcessFPPrecision) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - - case ISD::FPOW: - case ISD::FPOWI: { - // Promote f32 pow(i) to f64 pow(i). Note that this could insert a libcall - // directly as well, which may be better. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp2 = Node->getOperand(1); - if (Node->getOpcode() == ISD::FPOW) - Tmp2 = PromoteOp(Tmp2); - assert(Tmp1.getValueType() == NVT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - if (NoExcessFPPrecision) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - } - - case ISD::ATOMIC_CMP_SWAP: { - AtomicSDNode* AtomNode = cast<AtomicSDNode>(Node); - Tmp2 = PromoteOp(Node->getOperand(2)); - Tmp3 = PromoteOp(Node->getOperand(3)); - Result = DAG.getAtomic(Node->getOpcode(), dl, AtomNode->getMemoryVT(), - AtomNode->getChain(), - AtomNode->getBasePtr(), Tmp2, Tmp3, - AtomNode->getSrcValue(), - AtomNode->getAlignment()); - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Result.getValue(1))); - break; - } - case ISD::ATOMIC_LOAD_ADD: - case ISD::ATOMIC_LOAD_SUB: - case ISD::ATOMIC_LOAD_AND: - case ISD::ATOMIC_LOAD_OR: - case ISD::ATOMIC_LOAD_XOR: - case ISD::ATOMIC_LOAD_NAND: - case ISD::ATOMIC_LOAD_MIN: - case ISD::ATOMIC_LOAD_MAX: - case ISD::ATOMIC_LOAD_UMIN: - case ISD::ATOMIC_LOAD_UMAX: - case ISD::ATOMIC_SWAP: { - AtomicSDNode* AtomNode = cast<AtomicSDNode>(Node); - Tmp2 = PromoteOp(Node->getOperand(2)); - Result = DAG.getAtomic(Node->getOpcode(), dl, AtomNode->getMemoryVT(), - AtomNode->getChain(), - AtomNode->getBasePtr(), Tmp2, - AtomNode->getSrcValue(), - AtomNode->getAlignment()); - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Result.getValue(1))); - break; - } - - case ISD::AND: - case ISD::OR: - case ISD::XOR: - case ISD::ADD: - case ISD::SUB: - case ISD::MUL: - // The input may have strange things in the top bits of the registers, but - // these operations don't care. They may have weird bits going out, but - // that too is okay if they are integer operations. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp2 = PromoteOp(Node->getOperand(1)); - assert(Tmp1.getValueType() == NVT && Tmp2.getValueType() == NVT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - break; - case ISD::FADD: - case ISD::FSUB: - case ISD::FMUL: - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp2 = PromoteOp(Node->getOperand(1)); - assert(Tmp1.getValueType() == NVT && Tmp2.getValueType() == NVT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - - // Floating point operations will give excess precision that we may not be - // able to tolerate. If we DO allow excess precision, just leave it, - // otherwise excise it. - // FIXME: Why would we need to round FP ops more than integer ones? - // Is Round(Add(Add(A,B),C)) != Round(Add(Round(Add(A,B)), C)) - if (NoExcessFPPrecision) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - - case ISD::SDIV: - case ISD::SREM: - // These operators require that their input be sign extended. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp2 = PromoteOp(Node->getOperand(1)); - if (NVT.isInteger()) { - Tmp1 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Tmp1, - DAG.getValueType(VT)); - Tmp2 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Tmp2, - DAG.getValueType(VT)); - } - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - - // Perform FP_ROUND: this is probably overly pessimistic. - if (NVT.isFloatingPoint() && NoExcessFPPrecision) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - case ISD::FDIV: - case ISD::FREM: - case ISD::FCOPYSIGN: - // These operators require that their input be fp extended. - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "not implemented"); - case Legal: Tmp1 = LegalizeOp(Node->getOperand(0)); break; - case Promote: Tmp1 = PromoteOp(Node->getOperand(0)); break; - } - switch (getTypeAction(Node->getOperand(1).getValueType())) { - case Expand: assert(0 && "not implemented"); - case Legal: Tmp2 = LegalizeOp(Node->getOperand(1)); break; - case Promote: Tmp2 = PromoteOp(Node->getOperand(1)); break; - } - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - - // Perform FP_ROUND: this is probably overly pessimistic. - if (NoExcessFPPrecision && Node->getOpcode() != ISD::FCOPYSIGN) - Result = DAG.getNode(ISD::FP_ROUND_INREG, dl, NVT, Result, - DAG.getValueType(VT)); - break; - - case ISD::UDIV: - case ISD::UREM: - // These operators require that their input be zero extended. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp2 = PromoteOp(Node->getOperand(1)); - assert(NVT.isInteger() && "Operators don't apply to FP!"); - Tmp1 = DAG.getZeroExtendInReg(Tmp1, dl, VT); - Tmp2 = DAG.getZeroExtendInReg(Tmp2, dl, VT); - Result = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1, Tmp2); - break; - - case ISD::SHL: - Tmp1 = PromoteOp(Node->getOperand(0)); - Result = DAG.getNode(ISD::SHL, dl, NVT, Tmp1, Node->getOperand(1)); - break; - case ISD::SRA: - // The input value must be properly sign extended. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp1 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Tmp1, - DAG.getValueType(VT)); - Result = DAG.getNode(ISD::SRA, dl, NVT, Tmp1, Node->getOperand(1)); - break; - case ISD::SRL: - // The input value must be properly zero extended. - Tmp1 = PromoteOp(Node->getOperand(0)); - Tmp1 = DAG.getZeroExtendInReg(Tmp1, dl, VT); - Result = DAG.getNode(ISD::SRL, dl, NVT, Tmp1, Node->getOperand(1)); - break; - - case ISD::VAARG: - Tmp1 = Node->getOperand(0); // Get the chain. - Tmp2 = Node->getOperand(1); // Get the pointer. - if (TLI.getOperationAction(ISD::VAARG, VT) == TargetLowering::Custom) { - Tmp3 = DAG.getVAArg(VT, dl, Tmp1, Tmp2, Node->getOperand(2)); - Result = TLI.LowerOperation(Tmp3, DAG); - } else { - const Value *V = cast<SrcValueSDNode>(Node->getOperand(2))->getValue(); - SDValue VAList = DAG.getLoad(TLI.getPointerTy(), dl, Tmp1, Tmp2, V, 0); - // Increment the pointer, VAList, to the next vaarg - Tmp3 = DAG.getNode(ISD::ADD, dl, TLI.getPointerTy(), VAList, - DAG.getConstant(VT.getSizeInBits()/8, - TLI.getPointerTy())); - // Store the incremented VAList to the legalized pointer - Tmp3 = DAG.getStore(VAList.getValue(1), dl, Tmp3, Tmp2, V, 0); - // Load the actual argument out of the pointer VAList - Result = DAG.getExtLoad(ISD::EXTLOAD, dl, NVT, Tmp3, VAList, NULL, 0, VT); - } - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Result.getValue(1))); - break; - - case ISD::LOAD: { - LoadSDNode *LD = cast<LoadSDNode>(Node); - ISD::LoadExtType ExtType = ISD::isNON_EXTLoad(Node) - ? ISD::EXTLOAD : LD->getExtensionType(); - Result = DAG.getExtLoad(ExtType, dl, NVT, - LD->getChain(), LD->getBasePtr(), - LD->getSrcValue(), LD->getSrcValueOffset(), - LD->getMemoryVT(), - LD->isVolatile(), - LD->getAlignment()); - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Result.getValue(1))); - break; - } - case ISD::SELECT: { - Tmp2 = PromoteOp(Node->getOperand(1)); // Legalize the op0 - Tmp3 = PromoteOp(Node->getOperand(2)); // Legalize the op1 - - MVT VT2 = Tmp2.getValueType(); - assert(VT2 == Tmp3.getValueType() - && "PromoteOp SELECT: Operands 2 and 3 ValueTypes don't match"); - // Ensure that the resulting node is at least the same size as the operands' - // value types, because we cannot assume that TLI.getSetCCValueType() is - // constant. - Result = DAG.getNode(ISD::SELECT, dl, VT2, Node->getOperand(0), Tmp2, Tmp3); - break; - } - case ISD::SELECT_CC: - Tmp2 = PromoteOp(Node->getOperand(2)); // True - Tmp3 = PromoteOp(Node->getOperand(3)); // False - Result = DAG.getNode(ISD::SELECT_CC, dl, NVT, Node->getOperand(0), - Node->getOperand(1), Tmp2, Tmp3, Node->getOperand(4)); - break; - case ISD::BSWAP: - Tmp1 = Node->getOperand(0); - Tmp1 = DAG.getNode(ISD::ZERO_EXTEND, dl, NVT, Tmp1); - Tmp1 = DAG.getNode(ISD::BSWAP, dl, NVT, Tmp1); - Result = DAG.getNode(ISD::SRL, dl, NVT, Tmp1, - DAG.getConstant(NVT.getSizeInBits() - - VT.getSizeInBits(), - TLI.getShiftAmountTy())); - break; - case ISD::CTPOP: - case ISD::CTTZ: - case ISD::CTLZ: - // Zero extend the argument - Tmp1 = DAG.getNode(ISD::ZERO_EXTEND, dl, NVT, Node->getOperand(0)); - // Perform the larger operation, then subtract if needed. - Tmp1 = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1); - switch(Node->getOpcode()) { - case ISD::CTPOP: - Result = Tmp1; - break; - case ISD::CTTZ: - // if Tmp1 == sizeinbits(NVT) then Tmp1 = sizeinbits(Old VT) - Tmp2 = DAG.getSetCC(dl, TLI.getSetCCResultType(Tmp1.getValueType()), Tmp1, - DAG.getConstant(NVT.getSizeInBits(), NVT), - ISD::SETEQ); - Result = DAG.getNode(ISD::SELECT, dl, NVT, Tmp2, - DAG.getConstant(VT.getSizeInBits(), NVT), Tmp1); - break; - case ISD::CTLZ: - //Tmp1 = Tmp1 - (sizeinbits(NVT) - sizeinbits(Old VT)) - Result = DAG.getNode(ISD::SUB, dl, NVT, Tmp1, - DAG.getConstant(NVT.getSizeInBits() - - VT.getSizeInBits(), NVT)); - break; - } - break; - case ISD::EXTRACT_SUBVECTOR: - Result = PromoteOp(ExpandEXTRACT_SUBVECTOR(Op)); - break; - case ISD::EXTRACT_VECTOR_ELT: - Result = PromoteOp(ExpandEXTRACT_VECTOR_ELT(Op)); - break; - } - - assert(Result.getNode() && "Didn't set a result!"); - - // Make sure the result is itself legal. - Result = LegalizeOp(Result); - - // Remember that we promoted this! - AddPromotedOperand(Op, Result); - return Result; -} - /// ExpandEXTRACT_VECTOR_ELT - Expand an EXTRACT_VECTOR_ELT operation into /// a legal EXTRACT_VECTOR_ELT operation, scalar code, or memory traffic, /// based on the vector type. The return type of this matches the element type @@ -5027,12 +3786,9 @@ break; } case TargetLowering::Legal: - if (isTypeLegal(TVT)) { - Vec = LegalizeOp(Vec); - Op = DAG.UpdateNodeOperands(Op, Vec, Idx); - return Op; - } - break; + Vec = LegalizeOp(Vec); + Op = DAG.UpdateNodeOperands(Op, Vec, Idx); + return Op; case TargetLowering::Promote: assert(TVT.isVector() && "not vector type"); // fall thru to expand since vectors are by default are promote @@ -5042,23 +3798,7 @@ if (NumElems == 1) { // This must be an access of the only element. Return it. - Op = ScalarizeVectorOp(Vec); - } else if (!TLI.isTypeLegal(TVT) && isa<ConstantSDNode>(Idx)) { - unsigned NumLoElts = 1 << Log2_32(NumElems-1); - ConstantSDNode *CIdx = cast<ConstantSDNode>(Idx); - SDValue Lo, Hi; - SplitVectorOp(Vec, Lo, Hi); - if (CIdx->getZExtValue() < NumLoElts) { - Vec = Lo; - } else { - Vec = Hi; - Idx = DAG.getConstant(CIdx->getZExtValue() - NumLoElts, - Idx.getValueType()); - } - - // It's now an extract from the appropriate high or low part. Recurse. - Op = DAG.UpdateNodeOperands(Op, Vec, Idx); - Op = ExpandEXTRACT_VECTOR_ELT(Op); + Op = DAG.getNode(ISD::BIT_CONVERT, dl, Op.getValueType(), Vec); } else { // Store the value to a temporary stack slot, then LOAD the scalar // element back out. @@ -5082,305 +3822,8 @@ return Op; } -/// ExpandEXTRACT_SUBVECTOR - Expand a EXTRACT_SUBVECTOR operation. For now -/// we assume the operation can be split if it is not already legal. -SDValue SelectionDAGLegalize::ExpandEXTRACT_SUBVECTOR(SDValue Op) { - // We know that operand #0 is the Vec vector. For now we assume the index - // is a constant and that the extracted result is a supported hardware type. - SDValue Vec = Op.getOperand(0); - SDValue Idx = LegalizeOp(Op.getOperand(1)); - - unsigned NumElems = Vec.getValueType().getVectorNumElements(); - - if (NumElems == Op.getValueType().getVectorNumElements()) { - // This must be an access of the desired vector length. Return it. - return Vec; - } - - ConstantSDNode *CIdx = cast<ConstantSDNode>(Idx); - SDValue Lo, Hi; - SplitVectorOp(Vec, Lo, Hi); - if (CIdx->getZExtValue() < NumElems/2) { - Vec = Lo; - } else { - Vec = Hi; - Idx = DAG.getConstant(CIdx->getZExtValue() - NumElems/2, - Idx.getValueType()); - } - - // It's now an extract from the appropriate high or low part. Recurse. - Op = DAG.UpdateNodeOperands(Op, Vec, Idx); - return ExpandEXTRACT_SUBVECTOR(Op); -} - -/// LegalizeSetCCOperands - Attempts to create a legal LHS and RHS for a SETCC -/// with condition CC on the current target. This usually involves legalizing -/// or promoting the arguments. In the case where LHS and RHS must be expanded, -/// there may be no choice but to create a new SetCC node to represent the -/// legalized value of setcc lhs, rhs. In this case, the value is returned in -/// LHS, and the SDValue returned in RHS has a nil SDNode value. -void SelectionDAGLegalize::LegalizeSetCCOperands(SDValue &LHS, - SDValue &RHS, - SDValue &CC, - DebugLoc dl) { - SDValue Tmp1, Tmp2, Tmp3, Result; - - switch (getTypeAction(LHS.getValueType())) { - case Legal: - Tmp1 = LegalizeOp(LHS); // LHS - Tmp2 = LegalizeOp(RHS); // RHS - break; - case Promote: - Tmp1 = PromoteOp(LHS); // LHS - Tmp2 = PromoteOp(RHS); // RHS - - // If this is an FP compare, the operands have already been extended. - if (LHS.getValueType().isInteger()) { - MVT VT = LHS.getValueType(); - MVT NVT = TLI.getTypeToTransformTo(VT); - - // Otherwise, we have to insert explicit sign or zero extends. Note - // that we could insert sign extends for ALL conditions, but zero extend - // is cheaper on many machines (an AND instead of two shifts), so prefer - // it. - switch (cast<CondCodeSDNode>(CC)->get()) { - default: assert(0 && "Unknown integer comparison!"); - case ISD::SETEQ: - case ISD::SETNE: - case ISD::SETUGE: - case ISD::SETUGT: - case ISD::SETULE: - case ISD::SETULT: - // ALL of these operations will work if we either sign or zero extend - // the operands (including the unsigned comparisons!). Zero extend is - // usually a simpler/cheaper operation, so prefer it. - Tmp1 = DAG.getZeroExtendInReg(Tmp1, dl, VT); - Tmp2 = DAG.getZeroExtendInReg(Tmp2, dl, VT); - break; - case ISD::SETGE: - case ISD::SETGT: - case ISD::SETLT: - case ISD::SETLE: - Tmp1 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Tmp1, - DAG.getValueType(VT)); - Tmp2 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Tmp2, - DAG.getValueType(VT)); - Tmp1 = LegalizeOp(Tmp1); // Relegalize new nodes. - Tmp2 = LegalizeOp(Tmp2); // Relegalize new nodes. - break; - } - } - break; - case Expand: { - MVT VT = LHS.getValueType(); - if (VT == MVT::f32 || VT == MVT::f64) { - // Expand into one or more soft-fp libcall(s). - RTLIB::Libcall LC1 = RTLIB::UNKNOWN_LIBCALL, LC2 = RTLIB::UNKNOWN_LIBCALL; - switch (cast<CondCodeSDNode>(CC)->get()) { - case ISD::SETEQ: - case ISD::SETOEQ: - LC1 = (VT == MVT::f32) ? RTLIB::OEQ_F32 : RTLIB::OEQ_F64; - break; - case ISD::SETNE: - case ISD::SETUNE: - LC1 = (VT == MVT::f32) ? RTLIB::UNE_F32 : RTLIB::UNE_F64; - break; - case ISD::SETGE: - case ISD::SETOGE: - LC1 = (VT == MVT::f32) ? RTLIB::OGE_F32 : RTLIB::OGE_F64; - break; - case ISD::SETLT: - case ISD::SETOLT: - LC1 = (VT == MVT::f32) ? RTLIB::OLT_F32 : RTLIB::OLT_F64; - break; - case ISD::SETLE: - case ISD::SETOLE: - LC1 = (VT == MVT::f32) ? RTLIB::OLE_F32 : RTLIB::OLE_F64; - break; - case ISD::SETGT: - case ISD::SETOGT: - LC1 = (VT == MVT::f32) ? RTLIB::OGT_F32 : RTLIB::OGT_F64; - break; - case ISD::SETUO: - LC1 = (VT == MVT::f32) ? RTLIB::UO_F32 : RTLIB::UO_F64; - break; - case ISD::SETO: - LC1 = (VT == MVT::f32) ? RTLIB::O_F32 : RTLIB::O_F64; - break; - default: - LC1 = (VT == MVT::f32) ? RTLIB::UO_F32 : RTLIB::UO_F64; - switch (cast<CondCodeSDNode>(CC)->get()) { - case ISD::SETONE: - // SETONE = SETOLT | SETOGT - LC1 = (VT == MVT::f32) ? RTLIB::OLT_F32 : RTLIB::OLT_F64; - // Fallthrough - case ISD::SETUGT: - LC2 = (VT == MVT::f32) ? RTLIB::OGT_F32 : RTLIB::OGT_F64; - break; - case ISD::SETUGE: - LC2 = (VT == MVT::f32) ? RTLIB::OGE_F32 : RTLIB::OGE_F64; - break; - case ISD::SETULT: - LC2 = (VT == MVT::f32) ? RTLIB::OLT_F32 : RTLIB::OLT_F64; - break; - case ISD::SETULE: - LC2 = (VT == MVT::f32) ? RTLIB::OLE_F32 : RTLIB::OLE_F64; - break; - case ISD::SETUEQ: - LC2 = (VT == MVT::f32) ? RTLIB::OEQ_F32 : RTLIB::OEQ_F64; - break; - default: assert(0 && "Unsupported FP setcc!"); - } - } - - SDValue Dummy; - SDValue Ops[2] = { LHS, RHS }; - Tmp1 = ExpandLibCall(LC1, DAG.getMergeValues(Ops, 2, dl).getNode(), - false /*sign irrelevant*/, Dummy); - Tmp2 = DAG.getConstant(0, MVT::i32); - CC = DAG.getCondCode(TLI.getCmpLibcallCC(LC1)); - if (LC2 != RTLIB::UNKNOWN_LIBCALL) { - Tmp1 = DAG.getNode(ISD::SETCC, dl, - TLI.getSetCCResultType(Tmp1.getValueType()), - Tmp1, Tmp2, CC); - LHS = ExpandLibCall(LC2, DAG.getMergeValues(Ops, 2, dl).getNode(), - false /*sign irrelevant*/, Dummy); - Tmp2 = DAG.getNode(ISD::SETCC, dl, - TLI.getSetCCResultType(LHS.getValueType()), LHS, - Tmp2, DAG.getCondCode(TLI.getCmpLibcallCC(LC2))); - Tmp1 = DAG.getNode(ISD::OR, dl, Tmp1.getValueType(), Tmp1, Tmp2); - Tmp2 = SDValue(); - } - LHS = LegalizeOp(Tmp1); - RHS = Tmp2; - return; - } - - SDValue LHSLo, LHSHi, RHSLo, RHSHi; - ExpandOp(LHS, LHSLo, LHSHi); - ExpandOp(RHS, RHSLo, RHSHi); - ISD::CondCode CCCode = cast<CondCodeSDNode>(CC)->get(); - - if (VT==MVT::ppcf128) { - // FIXME: This generated code sucks. We want to generate - // FCMPU crN, hi1, hi2 - // BNE crN, L: - // FCMPU crN, lo1, lo2 - // The following can be improved, but not that much. - Tmp1 = DAG.getSetCC(dl, TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, ISD::SETOEQ); - Tmp2 = DAG.getSetCC(dl, TLI.getSetCCResultType(LHSLo.getValueType()), - LHSLo, RHSLo, CCCode); - Tmp3 = DAG.getNode(ISD::AND, dl, Tmp1.getValueType(), Tmp1, Tmp2); - Tmp1 = DAG.getSetCC(dl, TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, ISD::SETUNE); - Tmp2 = DAG.getSetCC(dl, TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, CCCode); - Tmp1 = DAG.getNode(ISD::AND, dl, Tmp1.getValueType(), Tmp1, Tmp2); - Tmp1 = DAG.getNode(ISD::OR, dl, Tmp1.getValueType(), Tmp1, Tmp3); - Tmp2 = SDValue(); - break; - } - - switch (CCCode) { - case ISD::SETEQ: - case ISD::SETNE: - if (RHSLo == RHSHi) - if (ConstantSDNode *RHSCST = dyn_cast<ConstantSDNode>(RHSLo)) - if (RHSCST->isAllOnesValue()) { - // Comparison to -1. - Tmp1 = DAG.getNode(ISD::AND, dl,LHSLo.getValueType(), LHSLo, LHSHi); - Tmp2 = RHSLo; - break; - } - - Tmp1 = DAG.getNode(ISD::XOR, dl, LHSLo.getValueType(), LHSLo, RHSLo); - Tmp2 = DAG.getNode(ISD::XOR, dl, LHSLo.getValueType(), LHSHi, RHSHi); - Tmp1 = DAG.getNode(ISD::OR, dl, Tmp1.getValueType(), Tmp1, Tmp2); - Tmp2 = DAG.getConstant(0, Tmp1.getValueType()); - break; - default: - // If this is a comparison of the sign bit, just look at the top part. - // X > -1, x < 0 - if (ConstantSDNode *CST = dyn_cast<ConstantSDNode>(RHS)) - if ((cast<CondCodeSDNode>(CC)->get() == ISD::SETLT && - CST->isNullValue()) || // X < 0 - (cast<CondCodeSDNode>(CC)->get() == ISD::SETGT && - CST->isAllOnesValue())) { // X > -1 - Tmp1 = LHSHi; - Tmp2 = RHSHi; - break; - } - - // FIXME: This generated code sucks. - ISD::CondCode LowCC; - switch (CCCode) { - default: assert(0 && "Unknown integer setcc!"); - case ISD::SETLT: - case ISD::SETULT: LowCC = ISD::SETULT; break; - case ISD::SETGT: - case ISD::SETUGT: LowCC = ISD::SETUGT; break; - case ISD::SETLE: - case ISD::SETULE: LowCC = ISD::SETULE; break; - case ISD::SETGE: - case ISD::SETUGE: LowCC = ISD::SETUGE; break; - } - - // Tmp1 = lo(op1) < lo(op2) // Always unsigned comparison - // Tmp2 = hi(op1) < hi(op2) // Signedness depends on operands - // dest = hi(op1) == hi(op2) ? Tmp1 : Tmp2; - - // NOTE: on targets without efficient SELECT of bools, we can always use - // this identity: (B1 ? B2 : B3) --> (B1 & B2)|(!B1&B3) - TargetLowering::DAGCombinerInfo DagCombineInfo(DAG, false, true, NULL); - Tmp1 = TLI.SimplifySetCC(TLI.getSetCCResultType(LHSLo.getValueType()), - LHSLo, RHSLo, LowCC, false, DagCombineInfo, dl); - if (!Tmp1.getNode()) - Tmp1 = DAG.getSetCC(dl, TLI.getSetCCResultType(LHSLo.getValueType()), - LHSLo, RHSLo, LowCC); - Tmp2 = TLI.SimplifySetCC(TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, CCCode, false, DagCombineInfo, dl); - if (!Tmp2.getNode()) - Tmp2 = DAG.getNode(ISD::SETCC, dl, - TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, CC); - - ConstantSDNode *Tmp1C = dyn_cast<ConstantSDNode>(Tmp1.getNode()); - ConstantSDNode *Tmp2C = dyn_cast<ConstantSDNode>(Tmp2.getNode()); - if ((Tmp1C && Tmp1C->isNullValue()) || - (Tmp2C && Tmp2C->isNullValue() && - (CCCode == ISD::SETLE || CCCode == ISD::SETGE || - CCCode == ISD::SETUGE || CCCode == ISD::SETULE)) || - (Tmp2C && Tmp2C->getAPIntValue() == 1 && - (CCCode == ISD::SETLT || CCCode == ISD::SETGT || - CCCode == ISD::SETUGT || CCCode == ISD::SETULT))) { - // low part is known false, returns high part. - // For LE / GE, if high part is known false, ignore the low part. - // For LT / GT, if high part is known true, ignore the low part. - Tmp1 = Tmp2; - Tmp2 = SDValue(); - } else { - Result = TLI.SimplifySetCC(TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, ISD::SETEQ, false, - DagCombineInfo, dl); - if (!Result.getNode()) - Result=DAG.getSetCC(dl, TLI.getSetCCResultType(LHSHi.getValueType()), - LHSHi, RHSHi, ISD::SETEQ); - Result = LegalizeOp(DAG.getNode(ISD::SELECT, dl, Tmp1.getValueType(), - Result, Tmp1, Tmp2)); - Tmp1 = Result; - Tmp2 = SDValue(); - } - } - } - } - LHS = Tmp1; - RHS = Tmp2; -} - /// LegalizeSetCCCondCode - Legalize a SETCC with given LHS and RHS and -/// condition code CC on the current target. This routine assumes LHS and rHS -/// have already been legalized by LegalizeSetCCOperands. It expands SETCC with +/// condition code CC on the current target. TIt expands SETCC with /// illegal condition code into AND / OR of multiple SETCC values. void SelectionDAGLegalize::LegalizeSetCCCondCode(MVT VT, SDValue &LHS, SDValue &RHS, @@ -5650,177 +4093,7 @@ return DAG.getLoad(VT, dl, StoreChain, FIPtr, SV, 0); } -void SelectionDAGLegalize::ExpandShiftParts(unsigned NodeOp, - SDValue Op, SDValue Amt, - SDValue &Lo, SDValue &Hi, - DebugLoc dl) { - // Expand the subcomponents. - SDValue LHSL, LHSH; - ExpandOp(Op, LHSL, LHSH); - SDValue Ops[] = { LHSL, LHSH, Amt }; - MVT VT = LHSL.getValueType(); - Lo = DAG.getNode(NodeOp, dl, DAG.getVTList(VT, VT), Ops, 3); - Hi = Lo.getValue(1); -} - - -/// ExpandShift - Try to find a clever way to expand this shift operation out to -/// smaller elements. If we can't find a way that is more efficient than a -/// libcall on this target, return false. Otherwise, return true with the -/// low-parts expanded into Lo and Hi. -bool SelectionDAGLegalize::ExpandShift(unsigned Opc, SDValue Op, SDValue Amt, - SDValue &Lo, SDValue &Hi, - DebugLoc dl) { - assert((Opc == ISD::SHL || Opc == ISD::SRA || Opc == ISD::SRL) && - "This is not a shift!"); - - MVT NVT = TLI.getTypeToTransformTo(Op.getValueType()); - SDValue ShAmt = LegalizeOp(Amt); - MVT ShTy = ShAmt.getValueType(); - unsigned ShBits = ShTy.getSizeInBits(); - unsigned VTBits = Op.getValueType().getSizeInBits(); - unsigned NVTBits = NVT.getSizeInBits(); - - // Handle the case when Amt is an immediate. - if (ConstantSDNode *CN = dyn_cast<ConstantSDNode>(Amt.getNode())) { - unsigned Cst = CN->getZExtValue(); - // Expand the incoming operand to be shifted, so that we have its parts - SDValue InL, InH; - ExpandOp(Op, InL, InH); - switch(Opc) { - case ISD::SHL: - if (Cst > VTBits) { - Lo = DAG.getConstant(0, NVT); - Hi = DAG.getConstant(0, NVT); - } else if (Cst > NVTBits) { - Lo = DAG.getConstant(0, NVT); - Hi = DAG.getNode(ISD::SHL, dl, - NVT, InL, DAG.getConstant(Cst-NVTBits, ShTy)); - } else if (Cst == NVTBits) { - Lo = DAG.getConstant(0, NVT); - Hi = InL; - } else { - Lo = DAG.getNode(ISD::SHL, dl, NVT, InL, DAG.getConstant(Cst, ShTy)); - Hi = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SHL, dl, NVT, InH, DAG.getConstant(Cst, ShTy)), - DAG.getNode(ISD::SRL, dl, NVT, InL, - DAG.getConstant(NVTBits-Cst, ShTy))); - } - return true; - case ISD::SRL: - if (Cst > VTBits) { - Lo = DAG.getConstant(0, NVT); - Hi = DAG.getConstant(0, NVT); - } else if (Cst > NVTBits) { - Lo = DAG.getNode(ISD::SRL, dl, NVT, - InH, DAG.getConstant(Cst-NVTBits, ShTy)); - Hi = DAG.getConstant(0, NVT); - } else if (Cst == NVTBits) { - Lo = InH; - Hi = DAG.getConstant(0, NVT); - } else { - Lo = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SRL, dl, NVT, InL, DAG.getConstant(Cst, ShTy)), - DAG.getNode(ISD::SHL, dl, NVT, InH, - DAG.getConstant(NVTBits-Cst, ShTy))); - Hi = DAG.getNode(ISD::SRL, dl, NVT, InH, DAG.getConstant(Cst, ShTy)); - } - return true; - case ISD::SRA: - if (Cst > VTBits) { - Hi = Lo = DAG.getNode(ISD::SRA, dl, NVT, InH, - DAG.getConstant(NVTBits-1, ShTy)); - } else if (Cst > NVTBits) { - Lo = DAG.getNode(ISD::SRA, dl, NVT, InH, - DAG.getConstant(Cst-NVTBits, ShTy)); - Hi = DAG.getNode(ISD::SRA, dl, NVT, InH, - DAG.getConstant(NVTBits-1, ShTy)); - } else if (Cst == NVTBits) { - Lo = InH; - Hi = DAG.getNode(ISD::SRA, dl, NVT, InH, - DAG.getConstant(NVTBits-1, ShTy)); - } else { - Lo = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SRL, dl, NVT, InL, DAG.getConstant(Cst, ShTy)), - DAG.getNode(ISD::SHL, dl, - NVT, InH, DAG.getConstant(NVTBits-Cst, ShTy))); - Hi = DAG.getNode(ISD::SRA, dl, NVT, InH, DAG.getConstant(Cst, ShTy)); - } - return true; - } - } - - // Okay, the shift amount isn't constant. However, if we can tell that it is - // >= 32 or < 32, we can still simplify it, without knowing the actual value. - APInt Mask = APInt::getHighBitsSet(ShBits, ShBits - Log2_32(NVTBits)); - APInt KnownZero, KnownOne; - DAG.ComputeMaskedBits(Amt, Mask, KnownZero, KnownOne); - - // If we know that if any of the high bits of the shift amount are one, then - // we can do this as a couple of simple shifts. - if (KnownOne.intersects(Mask)) { - // Mask out the high bit, which we know is set. - Amt = DAG.getNode(ISD::AND, dl, Amt.getValueType(), Amt, - DAG.getConstant(~Mask, Amt.getValueType())); - - // Expand the incoming operand to be shifted, so that we have its parts - SDValue InL, InH; - ExpandOp(Op, InL, InH); - switch(Opc) { - case ISD::SHL: - Lo = DAG.getConstant(0, NVT); // Low part is zero. - Hi = DAG.getNode(ISD::SHL, dl, NVT, InL, Amt); // High part from Lo part. - return true; - case ISD::SRL: - Hi = DAG.getConstant(0, NVT); // Hi part is zero. - Lo = DAG.getNode(ISD::SRL, dl, NVT, InH, Amt); // Lo part from Hi part. - return true; - case ISD::SRA: - Hi = DAG.getNode(ISD::SRA, dl, NVT, InH, // Sign extend high part. - DAG.getConstant(NVTBits-1, Amt.getValueType())); - Lo = DAG.getNode(ISD::SRA, dl, NVT, InH, Amt); // Lo part from Hi part. - return true; - } - } - - // If we know that the high bits of the shift amount are all zero, then we can - // do this as a couple of simple shifts. - if ((KnownZero & Mask) == Mask) { - // Compute 32-amt. - SDValue Amt2 = DAG.getNode(ISD::SUB, dl, Amt.getValueType(), - DAG.getConstant(NVTBits, Amt.getValueType()), - Amt); - - // Expand the incoming operand to be shifted, so that we have its parts - SDValue InL, InH; - ExpandOp(Op, InL, InH); - switch(Opc) { - case ISD::SHL: - Lo = DAG.getNode(ISD::SHL, dl, NVT, InL, Amt); - Hi = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SHL, dl, NVT, InH, Amt), - DAG.getNode(ISD::SRL, dl, NVT, InL, Amt2)); - return true; - case ISD::SRL: - Hi = DAG.getNode(ISD::SRL, dl, NVT, InH, Amt); - Lo = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SRL, dl, NVT, InL, Amt), - DAG.getNode(ISD::SHL, dl, NVT, InH, Amt2)); - return true; - case ISD::SRA: - Hi = DAG.getNode(ISD::SRA, dl, NVT, InH, Amt); - Lo = DAG.getNode(ISD::OR, dl, NVT, - DAG.getNode(ISD::SRL, dl, NVT, InL, Amt), - DAG.getNode(ISD::SHL, dl, NVT, InH, Amt2)); - return true; - } - } - - return false; -} - - // ExpandLibCall - Expand a node into a call to a libcall. If the result value // does not fit into a register, return the lo part and set the hi part to the // by-reg argument. If it does fit into a single register, return the result @@ -5857,17 +4130,7 @@ // the LastCALLSEQ_END to the legalized version of the CALLSEQ_END node that // was added by LowerCallTo (guaranteeing proper serialization of calls). LegalizeOp(CallInfo.second); - SDValue Result; - switch (getTypeAction(CallInfo.first.getValueType())) { - default: assert(0 && "Unknown thing"); - case Legal: - Result = CallInfo.first; - break; - case Expand: - ExpandOp(CallInfo.first, Result, Hi); - break; - } - return Result; + return CallInfo.first; } /// LegalizeINT_TO_FP - Legalize a [US]INT_TO_FP operation. @@ -5877,8 +4140,7 @@ DebugLoc dl) { bool isCustom = false; SDValue Tmp1; - switch (getTypeAction(Op.getValueType())) { - case Legal: + { switch (TLI.getOperationAction(isSigned ? ISD::SINT_TO_FP : ISD::UINT_TO_FP, Op.getValueType())) { default: assert(0 && "Unknown operation action!"); @@ -5904,173 +4166,10 @@ Result = PromoteLegalINT_TO_FP(LegalizeOp(Op), DestTy, isSigned, dl); break; } - break; - case Expand: - Result = ExpandIntToFP(isSigned, DestTy, Op, dl) ; - break; - case Promote: - Tmp1 = PromoteOp(Op); - if (isSigned) { - Tmp1 = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, Tmp1.getValueType(), - Tmp1, DAG.getValueType(Op.getValueType())); - } else { - Tmp1 = DAG.getZeroExtendInReg(Tmp1, dl, Op.getValueType()); - } - if (Result.getNode()) - Result = DAG.UpdateNodeOperands(Result, Tmp1); - else - Result = DAG.getNode(isSigned ? ISD::SINT_TO_FP : ISD::UINT_TO_FP, dl, - DestTy, Tmp1); - Result = LegalizeOp(Result); // The 'op' is not necessarily legal! - break; } return Result; } -/// ExpandIntToFP - Expand a [US]INT_TO_FP operation. -/// -SDValue SelectionDAGLegalize:: -ExpandIntToFP(bool isSigned, MVT DestTy, SDValue Source, DebugLoc dl) { - MVT SourceVT = Source.getValueType(); - bool ExpandSource = getTypeAction(SourceVT) == Expand; - - // Expand unsupported int-to-fp vector casts by unrolling them. - if (DestTy.isVector()) { - if (!ExpandSource) - return LegalizeOp(UnrollVectorOp(Source)); - MVT DestEltTy = DestTy.getVectorElementType(); - if (DestTy.getVectorNumElements() == 1) { - SDValue Scalar = ScalarizeVectorOp(Source); - SDValue Result = LegalizeINT_TO_FP(SDValue(), isSigned, - DestEltTy, Scalar, dl); - return DAG.getNode(ISD::BUILD_VECTOR, dl, DestTy, Result); - } - SDValue Lo, Hi; - SplitVectorOp(Source, Lo, Hi); - MVT SplitDestTy = MVT::getVectorVT(DestEltTy, - DestTy.getVectorNumElements() / 2); - SDValue LoResult = LegalizeINT_TO_FP(SDValue(), isSigned, SplitDestTy, - Lo, dl); - SDValue HiResult = LegalizeINT_TO_FP(SDValue(), isSigned, SplitDestTy, - Hi, dl); - return LegalizeOp(DAG.getNode(ISD::CONCAT_VECTORS, dl, DestTy, LoResult, - HiResult)); - } - - // Special case for i32 source to take advantage of UINTTOFP_I32_F32, etc. - if (!isSigned && SourceVT != MVT::i32) { - // The integer value loaded will be incorrectly if the 'sign bit' of the - // incoming integer is set. To handle this, we dynamically test to see if - // it is set, and, if so, add a fudge factor. - SDValue Hi; - if (ExpandSource) { - SDValue Lo; - ExpandOp(Source, Lo, Hi); - Source = DAG.getNode(ISD::BUILD_PAIR, dl, SourceVT, Lo, Hi); - } else { - // The comparison for the sign bit will use the entire operand. - Hi = Source; - } - - // Check to see if the target has a custom way to lower this. If so, use - // it. (Note we've already expanded the operand in this case.) - switch (TLI.getOperationAction(ISD::UINT_TO_FP, SourceVT)) { - default: assert(0 && "This action not implemented for this operation!"); - case TargetLowering::Legal: - case TargetLowering::Expand: - break; // This case is handled below. - case TargetLowering::Custom: { - SDValue NV = TLI.LowerOperation(DAG.getNode(ISD::UINT_TO_FP, dl, DestTy, - Source), DAG); - if (NV.getNode()) - return LegalizeOp(NV); - break; // The target decided this was legal after all - } - } - - // If this is unsigned, and not supported, first perform the conversion to - // signed, then adjust the result if the sign bit is set. - SDValue SignedConv = ExpandIntToFP(true, DestTy, Source, dl); - - SDValue SignSet = DAG.getSetCC(dl, - TLI.getSetCCResultType(Hi.getValueType()), - Hi, DAG.getConstant(0, Hi.getValueType()), - ISD::SETLT); - SDValue Zero = DAG.getIntPtrConstant(0), Four = DAG.getIntPtrConstant(4); - SDValue CstOffset = DAG.getNode(ISD::SELECT, dl, Zero.getValueType(), - SignSet, Four, Zero); - uint64_t FF = 0x5f800000ULL; - if (TLI.isLittleEndian()) FF <<= 32; - Constant *FudgeFactor = ConstantInt::get(Type::Int64Ty, FF); - - SDValue CPIdx = DAG.getConstantPool(FudgeFactor, TLI.getPointerTy()); - unsigned Alignment = cast<ConstantPoolSDNode>(CPIdx)->getAlignment(); - CPIdx = DAG.getNode(ISD::ADD, dl, TLI.getPointerTy(), CPIdx, CstOffset); - Alignment = std::min(Alignment, 4u); - SDValue FudgeInReg; - if (DestTy == MVT::f32) - FudgeInReg = DAG.getLoad(MVT::f32, dl, DAG.getEntryNode(), CPIdx, - PseudoSourceValue::getConstantPool(), 0, - false, Alignment); - else if (DestTy.bitsGT(MVT::f32)) - // FIXME: Avoid the extend by construction the right constantpool? - FudgeInReg = DAG.getExtLoad(ISD::EXTLOAD, dl, DestTy, DAG.getEntryNode(), - CPIdx, - PseudoSourceValue::getConstantPool(), 0, - MVT::f32, false, Alignment); - else - assert(0 && "Unexpected conversion"); - - MVT SCVT = SignedConv.getValueType(); - if (SCVT != DestTy) { - // Destination type needs to be expanded as well. The FADD now we are - // constructing will be expanded into a libcall. - if (SCVT.getSizeInBits() != DestTy.getSizeInBits()) { - assert(SCVT.getSizeInBits() * 2 == DestTy.getSizeInBits()); - SignedConv = DAG.getNode(ISD::BUILD_PAIR, dl, DestTy, - SignedConv, SignedConv.getValue(1)); - } - SignedConv = DAG.getNode(ISD::BIT_CONVERT, dl, DestTy, SignedConv); - } - return DAG.getNode(ISD::FADD, dl, DestTy, SignedConv, FudgeInReg); - } - - // Check to see if the target has a custom way to lower this. If so, use it. - switch (TLI.getOperationAction(ISD::SINT_TO_FP, SourceVT)) { - default: assert(0 && "This action not implemented for this operation!"); - case TargetLowering::Legal: - case TargetLowering::Expand: - break; // This case is handled below. - case TargetLowering::Custom: { - SDValue NV = TLI.LowerOperation(DAG.getNode(ISD::SINT_TO_FP, dl, DestTy, - Source), DAG); - if (NV.getNode()) - return LegalizeOp(NV); - break; // The target decided this was legal after all - } - } - - // Expand the source, then glue it back together for the call. We must expand - // the source in case it is shared (this pass of legalize must traverse it). - if (ExpandSource) { - SDValue SrcLo, SrcHi; - ExpandOp(Source, SrcLo, SrcHi); - Source = DAG.getNode(ISD::BUILD_PAIR, dl, SourceVT, SrcLo, SrcHi); - } - - RTLIB::Libcall LC = isSigned ? - RTLIB::getSINTTOFP(SourceVT, DestTy) : - RTLIB::getUINTTOFP(SourceVT, DestTy); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unknown int value type"); - - Source = DAG.getNode(ISD::SINT_TO_FP, dl, DestTy, Source); - SDValue HiPart; - SDValue Result = ExpandLibCall(LC, Source.getNode(), isSigned, HiPart); - if (Result.getValueType() != DestTy && HiPart.getNode()) - Result = DAG.getNode(ISD::BUILD_PAIR, dl, DestTy, Result, HiPart); - return Result; -} - /// ExpandLegalINT_TO_FP - This function is responsible for legalizing a /// INT_TO_FP operation of the specified operand when the target requests that /// we expand it. At this point, we know that the result and operand types are @@ -6418,2201 +4517,6 @@ } } -/// ExpandOp - Expand the specified SDValue into its two component pieces -/// Lo&Hi. Note that the Op MUST be an expanded type. As a result of this, the -/// LegalizedNodes map is filled in for any results that are not expanded, the -/// ExpandedNodes map is filled in for any results that are expanded, and the -/// Lo/Hi values are returned. -void SelectionDAGLegalize::ExpandOp(SDValue Op, SDValue &Lo, SDValue &Hi){ - MVT VT = Op.getValueType(); - MVT NVT = TLI.getTypeToTransformTo(VT); - SDNode *Node = Op.getNode(); - DebugLoc dl = Node->getDebugLoc(); - assert(getTypeAction(VT) == Expand && "Not an expanded type!"); - assert(((NVT.isInteger() && NVT.bitsLT(VT)) || VT.isFloatingPoint() || - VT.isVector()) && "Cannot expand to FP value or to larger int value!"); - - // See if we already expanded it. - DenseMap<SDValue, std::pair<SDValue, SDValue> >::iterator I - = ExpandedNodes.find(Op); - if (I != ExpandedNodes.end()) { - Lo = I->second.first; - Hi = I->second.second; - return; - } - - switch (Node->getOpcode()) { - case ISD::CopyFromReg: - assert(0 && "CopyFromReg must be legal!"); - case ISD::FP_ROUND_INREG: - if (VT == MVT::ppcf128 && - TLI.getOperationAction(ISD::FP_ROUND_INREG, VT) =- TargetLowering::Custom) { - SDValue SrcLo, SrcHi, Src; - ExpandOp(Op.getOperand(0), SrcLo, SrcHi); - Src = DAG.getNode(ISD::BUILD_PAIR, dl, VT, SrcLo, SrcHi); - SDValue Result - TLI.LowerOperation(DAG.getNode(ISD::FP_ROUND_INREG, dl, VT, Src, - Op.getOperand(1)), DAG); - assert(Result.getNode()->getOpcode() == ISD::BUILD_PAIR); - Lo = Result.getNode()->getOperand(0); - Hi = Result.getNode()->getOperand(1); - break; - } - // fall through - default: -#ifndef NDEBUG - cerr << "NODE: "; Node->dump(&DAG); cerr << "\n"; -#endif - assert(0 && "Do not know how to expand this operator!"); - abort(); - case ISD::EXTRACT_ELEMENT: - ExpandOp(Node->getOperand(0), Lo, Hi); - if (cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue()) - return ExpandOp(Hi, Lo, Hi); - return ExpandOp(Lo, Lo, Hi); - case ISD::EXTRACT_VECTOR_ELT: - // ExpandEXTRACT_VECTOR_ELT tolerates invalid result types. - Lo = ExpandEXTRACT_VECTOR_ELT(Op); - return ExpandOp(Lo, Lo, Hi); - case ISD::UNDEF: - Lo = DAG.getUNDEF(NVT); - Hi = DAG.getUNDEF(NVT); - break; - case ISD::Constant: { - unsigned NVTBits = NVT.getSizeInBits(); - const APInt &Cst = cast<ConstantSDNode>(Node)->getAPIntValue(); - Lo = DAG.getConstant(APInt(Cst).trunc(NVTBits), NVT); - Hi = DAG.getConstant(Cst.lshr(NVTBits).trunc(NVTBits), NVT); - break; - } - case ISD::ConstantFP: { - ConstantFPSDNode *CFP = cast<ConstantFPSDNode>(Node); - if (CFP->getValueType(0) == MVT::ppcf128) { - APInt api = CFP->getValueAPF().bitcastToAPInt(); - Lo = DAG.getConstantFP(APFloat(APInt(64, 1, &api.getRawData()[1])), - MVT::f64); - Hi = DAG.getConstantFP(APFloat(APInt(64, 1, &api.getRawData()[0])), - MVT::f64); - break; - } - Lo = ExpandConstantFP(CFP, false, DAG, TLI); - if (getTypeAction(Lo.getValueType()) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - case ISD::BUILD_PAIR: - // Return the operands. - Lo = Node->getOperand(0); - Hi = Node->getOperand(1); - break; - - case ISD::MERGE_VALUES: - if (Node->getNumValues() == 1) { - ExpandOp(Op.getOperand(0), Lo, Hi); - break; - } - // FIXME: For now only expand i64,chain = MERGE_VALUES (x, y) - assert(Op.getResNo() == 0 && Node->getNumValues() == 2 && - Op.getValue(1).getValueType() == MVT::Other && - "unhandled MERGE_VALUES"); - ExpandOp(Op.getOperand(0), Lo, Hi); - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Op.getOperand(1))); - break; - - case ISD::SIGN_EXTEND_INREG: - ExpandOp(Node->getOperand(0), Lo, Hi); - // sext_inreg the low part if needed. - Lo = DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, NVT, Lo, Node->getOperand(1)); - - // The high part gets the sign extension from the lo-part. This handles - // things like sextinreg V:i64 from i8. - Hi = DAG.getNode(ISD::SRA, dl, NVT, Lo, - DAG.getConstant(NVT.getSizeInBits()-1, - TLI.getShiftAmountTy())); - break; - - case ISD::BSWAP: { - ExpandOp(Node->getOperand(0), Lo, Hi); - SDValue TempLo = DAG.getNode(ISD::BSWAP, dl, NVT, Hi); - Hi = DAG.getNode(ISD::BSWAP, dl, NVT, Lo); - Lo = TempLo; - break; - } - - case ISD::CTPOP: - ExpandOp(Node->getOperand(0), Lo, Hi); - Lo = DAG.getNode(ISD::ADD, dl, NVT, // ctpop(HL) -> ctpop(H)+ctpop(L) - DAG.getNode(ISD::CTPOP, dl, NVT, Lo), - DAG.getNode(ISD::CTPOP, dl, NVT, Hi)); - Hi = DAG.getConstant(0, NVT); - break; - - case ISD::CTLZ: { - // ctlz (HL) -> ctlz(H) != 32 ? ctlz(H) : (ctlz(L)+32) - ExpandOp(Node->getOperand(0), Lo, Hi); - SDValue BitsC = DAG.getConstant(NVT.getSizeInBits(), NVT); - SDValue HLZ = DAG.getNode(ISD::CTLZ, dl, NVT, Hi); - SDValue TopNotZero = DAG.getSetCC(dl, TLI.getSetCCResultType(NVT), HLZ, - BitsC, ISD::SETNE); - SDValue LowPart = DAG.getNode(ISD::CTLZ, dl, NVT, Lo); - LowPart = DAG.getNode(ISD::ADD, dl, NVT, LowPart, BitsC); - - Lo = DAG.getNode(ISD::SELECT, dl, NVT, TopNotZero, HLZ, LowPart); - Hi = DAG.getConstant(0, NVT); - break; - } - - case ISD::CTTZ: { - // cttz (HL) -> cttz(L) != 32 ? cttz(L) : (cttz(H)+32) - ExpandOp(Node->getOperand(0), Lo, Hi); - SDValue BitsC = DAG.getConstant(NVT.getSizeInBits(), NVT); - SDValue LTZ = DAG.getNode(ISD::CTTZ, dl, NVT, Lo); - SDValue BotNotZero = DAG.getSetCC(dl, TLI.getSetCCResultType(NVT), LTZ, - BitsC, ISD::SETNE); - SDValue HiPart = DAG.getNode(ISD::CTTZ, dl, NVT, Hi); - HiPart = DAG.getNode(ISD::ADD, dl, NVT, HiPart, BitsC); - - Lo = DAG.getNode(ISD::SELECT, dl, NVT, BotNotZero, LTZ, HiPart); - Hi = DAG.getConstant(0, NVT); - break; - } - - case ISD::VAARG: { - SDValue Ch = Node->getOperand(0); // Legalize the chain. - SDValue Ptr = Node->getOperand(1); // Legalize the pointer. - Lo = DAG.getVAArg(NVT, dl, Ch, Ptr, Node->getOperand(2)); - Hi = DAG.getVAArg(NVT, dl, Lo.getValue(1), Ptr, Node->getOperand(2)); - - // Remember that we legalized the chain. - Hi = LegalizeOp(Hi); - AddLegalizedOperand(Op.getValue(1), Hi.getValue(1)); - if (TLI.isBigEndian()) - std::swap(Lo, Hi); - break; - } - - case ISD::LOAD: { - LoadSDNode *LD = cast<LoadSDNode>(Node); - SDValue Ch = LD->getChain(); // Legalize the chain. - SDValue Ptr = LD->getBasePtr(); // Legalize the pointer. - ISD::LoadExtType ExtType = LD->getExtensionType(); - const Value *SV = LD->getSrcValue(); - int SVOffset = LD->getSrcValueOffset(); - unsigned Alignment = LD->getAlignment(); - bool isVolatile = LD->isVolatile(); - - if (ExtType == ISD::NON_EXTLOAD) { - Lo = DAG.getLoad(NVT, dl, Ch, Ptr, SV, SVOffset, - isVolatile, Alignment); - if (VT == MVT::f32 || VT == MVT::f64) { - // f32->i32 or f64->i64 one to one expansion. - // Remember that we legalized the chain. - AddLegalizedOperand(SDValue(Node, 1), LegalizeOp(Lo.getValue(1))); - // Recursively expand the new load. - if (getTypeAction(NVT) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - - // Increment the pointer to the other half. - unsigned IncrementSize = Lo.getValueType().getSizeInBits()/8; - Ptr = DAG.getNode(ISD::ADD, dl, Ptr.getValueType(), Ptr, - DAG.getIntPtrConstant(IncrementSize)); - SVOffset += IncrementSize; - Alignment = MinAlign(Alignment, IncrementSize); - Hi = DAG.getLoad(NVT, dl, Ch, Ptr, SV, SVOffset, - isVolatile, Alignment); - - // Build a factor node to remember that this load is independent of the - // other one. - SDValue TF = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1), - Hi.getValue(1)); - - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(TF)); - if (TLI.isBigEndian()) - std::swap(Lo, Hi); - } else { - MVT EVT = LD->getMemoryVT(); - - if ((VT == MVT::f64 && EVT == MVT::f32) || - (VT == MVT::ppcf128 && (EVT==MVT::f64 || EVT==MVT::f32))) { - // f64 = EXTLOAD f32 should expand to LOAD, FP_EXTEND - SDValue Load = DAG.getLoad(EVT, dl, Ch, Ptr, SV, - SVOffset, isVolatile, Alignment); - // Remember that we legalized the chain. - AddLegalizedOperand(SDValue(Node, 1), LegalizeOp(Load.getValue(1))); - ExpandOp(DAG.getNode(ISD::FP_EXTEND, dl, VT, Load), Lo, Hi); - break; - } - - if (EVT == NVT) - Lo = DAG.getLoad(NVT, dl, Ch, Ptr, SV, - SVOffset, isVolatile, Alignment); - else - Lo = DAG.getExtLoad(ExtType, dl, NVT, Ch, Ptr, SV, - SVOffset, EVT, isVolatile, - Alignment); - - // Remember that we legalized the chain. - AddLegalizedOperand(SDValue(Node, 1), LegalizeOp(Lo.getValue(1))); - - if (ExtType == ISD::SEXTLOAD) { - // The high part is obtained by SRA'ing all but one of the bits of the - // lo part. - unsigned LoSize = Lo.getValueType().getSizeInBits(); - Hi = DAG.getNode(ISD::SRA, dl, NVT, Lo, - DAG.getConstant(LoSize-1, TLI.getShiftAmountTy())); - } else if (ExtType == ISD::ZEXTLOAD) { - // The high part is just a zero. - Hi = DAG.getConstant(0, NVT); - } else /* if (ExtType == ISD::EXTLOAD) */ { - // The high part is undefined. - Hi = DAG.getUNDEF(NVT); - } - } - break; - } - case ISD::AND: - case ISD::OR: - case ISD::XOR: { // Simple logical operators -> two trivial pieces. - SDValue LL, LH, RL, RH; - ExpandOp(Node->getOperand(0), LL, LH); - ExpandOp(Node->getOperand(1), RL, RH); - Lo = DAG.getNode(Node->getOpcode(), dl, NVT, LL, RL); - Hi = DAG.getNode(Node->getOpcode(), dl, NVT, LH, RH); - break; - } - case ISD::SELECT: { - SDValue LL, LH, RL, RH; - ExpandOp(Node->getOperand(1), LL, LH); - ExpandOp(Node->getOperand(2), RL, RH); - if (getTypeAction(NVT) == Expand) - NVT = TLI.getTypeToExpandTo(NVT); - Lo = DAG.getNode(ISD::SELECT, dl, NVT, Node->getOperand(0), LL, RL); - if (VT != MVT::f32) - Hi = DAG.getNode(ISD::SELECT, dl, NVT, Node->getOperand(0), LH, RH); - break; - } - case ISD::SELECT_CC: { - SDValue TL, TH, FL, FH; - ExpandOp(Node->getOperand(2), TL, TH); - ExpandOp(Node->getOperand(3), FL, FH); - if (getTypeAction(NVT) == Expand) - NVT = TLI.getTypeToExpandTo(NVT); - Lo = DAG.getNode(ISD::SELECT_CC, dl, NVT, Node->getOperand(0), - Node->getOperand(1), TL, FL, Node->getOperand(4)); - if (VT != MVT::f32) - Hi = DAG.getNode(ISD::SELECT_CC, dl, NVT, Node->getOperand(0), - Node->getOperand(1), TH, FH, Node->getOperand(4)); - break; - } - case ISD::ANY_EXTEND: - // The low part is any extension of the input (which degenerates to a copy). - Lo = DAG.getNode(ISD::ANY_EXTEND, dl, NVT, Node->getOperand(0)); - // The high part is undefined. - Hi = DAG.getUNDEF(NVT); - break; - case ISD::SIGN_EXTEND: { - // The low part is just a sign extension of the input (which degenerates to - // a copy). - Lo = DAG.getNode(ISD::SIGN_EXTEND, dl, NVT, Node->getOperand(0)); - - // The high part is obtained by SRA'ing all but one of the bits of the lo - // part. - unsigned LoSize = Lo.getValueType().getSizeInBits(); - Hi = DAG.getNode(ISD::SRA, dl, NVT, Lo, - DAG.getConstant(LoSize-1, TLI.getShiftAmountTy())); - break; - } - case ISD::ZERO_EXTEND: - // The low part is just a zero extension of the input (which degenerates to - // a copy). - Lo = DAG.getNode(ISD::ZERO_EXTEND, dl, NVT, Node->getOperand(0)); - - // The high part is just a zero. - Hi = DAG.getConstant(0, NVT); - break; - - case ISD::TRUNCATE: { - // The input value must be larger than this value. Expand *it*. - SDValue NewLo; - ExpandOp(Node->getOperand(0), NewLo, Hi); - - // The low part is now either the right size, or it is closer. If not the - // right size, make an illegal truncate so we recursively expand it. - if (NewLo.getValueType() != Node->getValueType(0)) - NewLo = DAG.getNode(ISD::TRUNCATE, dl, Node->getValueType(0), NewLo); - ExpandOp(NewLo, Lo, Hi); - break; - } - - case ISD::BIT_CONVERT: { - SDValue Tmp; - if (TLI.getOperationAction(ISD::BIT_CONVERT, VT) == TargetLowering::Custom){ - // If the target wants to, allow it to lower this itself. - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "cannot expand FP!"); - case Legal: Tmp = LegalizeOp(Node->getOperand(0)); break; - case Promote: Tmp = PromoteOp (Node->getOperand(0)); break; - } - Tmp = TLI.LowerOperation(DAG.getNode(ISD::BIT_CONVERT, dl, VT, Tmp), DAG); - } - - // f32 / f64 must be expanded to i32 / i64. - if (VT == MVT::f32 || VT == MVT::f64) { - Lo = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Node->getOperand(0)); - if (getTypeAction(NVT) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - - // If source operand will be expanded to the same type as VT, i.e. - // i64 <- f64, i32 <- f32, expand the source operand instead. - MVT VT0 = Node->getOperand(0).getValueType(); - if (getTypeAction(VT0) == Expand && TLI.getTypeToTransformTo(VT0) == VT) { - ExpandOp(Node->getOperand(0), Lo, Hi); - break; - } - - // Turn this into a load/store pair by default. - if (Tmp.getNode() == 0) - Tmp = EmitStackConvert(Node->getOperand(0), VT, VT, dl); - - ExpandOp(Tmp, Lo, Hi); - break; - } - - case ISD::READCYCLECOUNTER: { - assert(TLI.getOperationAction(ISD::READCYCLECOUNTER, VT) =- TargetLowering::Custom && - "Must custom expand ReadCycleCounter"); - SDValue Tmp = TLI.LowerOperation(Op, DAG); - assert(Tmp.getNode() && "Node must be custom expanded!"); - ExpandOp(Tmp.getValue(0), Lo, Hi); - AddLegalizedOperand(SDValue(Node, 1), // Remember we legalized the chain. - LegalizeOp(Tmp.getValue(1))); - break; - } - - case ISD::ATOMIC_CMP_SWAP: { - // This operation does not need a loop. - SDValue Tmp = TLI.LowerOperation(Op, DAG); - assert(Tmp.getNode() && "Node must be custom expanded!"); - ExpandOp(Tmp.getValue(0), Lo, Hi); - AddLegalizedOperand(SDValue(Node, 1), // Remember we legalized the chain. - LegalizeOp(Tmp.getValue(1))); - break; - } - - case ISD::ATOMIC_LOAD_ADD: - case ISD::ATOMIC_LOAD_SUB: - case ISD::ATOMIC_LOAD_AND: - case ISD::ATOMIC_LOAD_OR: - case ISD::ATOMIC_LOAD_XOR: - case ISD::ATOMIC_LOAD_NAND: - case ISD::ATOMIC_SWAP: { - // These operations require a loop to be generated. We can't do that yet, - // so substitute a target-dependent pseudo and expand that later. - SDValue In2Lo, In2Hi, In2; - ExpandOp(Op.getOperand(2), In2Lo, In2Hi); - In2 = DAG.getNode(ISD::BUILD_PAIR, dl, VT, In2Lo, In2Hi); - AtomicSDNode* Anode = cast<AtomicSDNode>(Node); - SDValue Replace - DAG.getAtomic(Op.getOpcode(), dl, Anode->getMemoryVT(), - Op.getOperand(0), Op.getOperand(1), In2, - Anode->getSrcValue(), Anode->getAlignment()); - SDValue Result = TLI.LowerOperation(Replace, DAG); - ExpandOp(Result.getValue(0), Lo, Hi); - // Remember that we legalized the chain. - AddLegalizedOperand(SDValue(Node, 1), LegalizeOp(Result.getValue(1))); - break; - } - - // These operators cannot be expanded directly, emit them as calls to - // library functions. - case ISD::FP_TO_SINT: { - if (TLI.getOperationAction(ISD::FP_TO_SINT, VT) == TargetLowering::Custom) { - SDValue Op; - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "cannot expand FP!"); - case Legal: Op = LegalizeOp(Node->getOperand(0)); break; - case Promote: Op = PromoteOp (Node->getOperand(0)); break; - } - - Op = TLI.LowerOperation(DAG.getNode(ISD::FP_TO_SINT, dl, VT, Op), DAG); - - // Now that the custom expander is done, expand the result, which is still - // VT. - if (Op.getNode()) { - ExpandOp(Op, Lo, Hi); - break; - } - } - - RTLIB::Libcall LC = RTLIB::getFPTOSINT(Node->getOperand(0).getValueType(), - VT); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unexpected uint-to-fp conversion!"); - Lo = ExpandLibCall(LC, Node, false/*sign irrelevant*/, Hi); - break; - } - - case ISD::FP_TO_UINT: { - if (TLI.getOperationAction(ISD::FP_TO_UINT, VT) == TargetLowering::Custom) { - SDValue Op; - switch (getTypeAction(Node->getOperand(0).getValueType())) { - case Expand: assert(0 && "cannot expand FP!"); - case Legal: Op = LegalizeOp(Node->getOperand(0)); break; - case Promote: Op = PromoteOp (Node->getOperand(0)); break; - } - - Op = TLI.LowerOperation(DAG.getNode(ISD::FP_TO_UINT, dl, VT, Op), DAG); - - // Now that the custom expander is done, expand the result. - if (Op.getNode()) { - ExpandOp(Op, Lo, Hi); - break; - } - } - - RTLIB::Libcall LC = RTLIB::getFPTOUINT(Node->getOperand(0).getValueType(), - VT); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unexpected fp-to-uint conversion!"); - Lo = ExpandLibCall(LC, Node, false/*sign irrelevant*/, Hi); - break; - } - - case ISD::SHL: { - // If the target wants custom lowering, do so. - SDValue ShiftAmt = LegalizeOp(Node->getOperand(1)); - if (TLI.getOperationAction(ISD::SHL, VT) == TargetLowering::Custom) { - SDValue Op = DAG.getNode(ISD::SHL, dl, VT, Node->getOperand(0), ShiftAmt); - Op = TLI.LowerOperation(Op, DAG); - if (Op.getNode()) { - // Now that the custom expander is done, expand the result, which is - // still VT. - ExpandOp(Op, Lo, Hi); - break; - } - } - - // If ADDC/ADDE are supported and if the shift amount is a constant 1, emit - // this X << 1 as X+X. - if (ConstantSDNode *ShAmt = dyn_cast<ConstantSDNode>(ShiftAmt)) { - if (ShAmt->getAPIntValue() == 1 && - TLI.isOperationLegalOrCustom(ISD::ADDC, NVT) && - TLI.isOperationLegalOrCustom(ISD::ADDE, NVT)) { - SDValue LoOps[2], HiOps[3]; - ExpandOp(Node->getOperand(0), LoOps[0], HiOps[0]); - SDVTList VTList = DAG.getVTList(LoOps[0].getValueType(), MVT::Flag); - LoOps[1] = LoOps[0]; - Lo = DAG.getNode(ISD::ADDC, dl, VTList, LoOps, 2); - - HiOps[1] = HiOps[0]; - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(ISD::ADDE, dl, VTList, HiOps, 3); - break; - } - } - - // If we can emit an efficient shift operation, do so now. - if (ExpandShift(ISD::SHL, Node->getOperand(0), ShiftAmt, Lo, Hi, dl)) - break; - - // If this target supports SHL_PARTS, use it. - TargetLowering::LegalizeAction Action - TLI.getOperationAction(ISD::SHL_PARTS, NVT); - if ((Action == TargetLowering::Legal && TLI.isTypeLegal(NVT)) || - Action == TargetLowering::Custom) { - ExpandShiftParts(ISD::SHL_PARTS, Node->getOperand(0), - ShiftAmt, Lo, Hi, dl); - break; - } - - // Otherwise, emit a libcall. - Lo = ExpandLibCall(RTLIB::SHL_I64, Node, false/*left shift=unsigned*/, Hi); - break; - } - - case ISD::SRA: { - // If the target wants custom lowering, do so. - SDValue ShiftAmt = LegalizeOp(Node->getOperand(1)); - if (TLI.getOperationAction(ISD::SRA, VT) == TargetLowering::Custom) { - SDValue Op = DAG.getNode(ISD::SRA, dl, VT, Node->getOperand(0), ShiftAmt); - Op = TLI.LowerOperation(Op, DAG); - if (Op.getNode()) { - // Now that the custom expander is done, expand the result, which is - // still VT. - ExpandOp(Op, Lo, Hi); - break; - } - } - - // If we can emit an efficient shift operation, do so now. - if (ExpandShift(ISD::SRA, Node->getOperand(0), ShiftAmt, Lo, Hi, dl)) - break; - - // If this target supports SRA_PARTS, use it. - TargetLowering::LegalizeAction Action - TLI.getOperationAction(ISD::SRA_PARTS, NVT); - if ((Action == TargetLowering::Legal && TLI.isTypeLegal(NVT)) || - Action == TargetLowering::Custom) { - ExpandShiftParts(ISD::SRA_PARTS, Node->getOperand(0), - ShiftAmt, Lo, Hi, dl); - break; - } - - // Otherwise, emit a libcall. - Lo = ExpandLibCall(RTLIB::SRA_I64, Node, true/*ashr is signed*/, Hi); - break; - } - - case ISD::SRL: { - // If the target wants custom lowering, do so. - SDValue ShiftAmt = LegalizeOp(Node->getOperand(1)); - if (TLI.getOperationAction(ISD::SRL, VT) == TargetLowering::Custom) { - SDValue Op = DAG.getNode(ISD::SRL, dl, VT, Node->getOperand(0), ShiftAmt); - Op = TLI.LowerOperation(Op, DAG); - if (Op.getNode()) { - // Now that the custom expander is done, expand the result, which is - // still VT. - ExpandOp(Op, Lo, Hi); - break; - } - } - - // If we can emit an efficient shift operation, do so now. - if (ExpandShift(ISD::SRL, Node->getOperand(0), ShiftAmt, Lo, Hi, dl)) - break; - - // If this target supports SRL_PARTS, use it. - TargetLowering::LegalizeAction Action - TLI.getOperationAction(ISD::SRL_PARTS, NVT); - if ((Action == TargetLowering::Legal && TLI.isTypeLegal(NVT)) || - Action == TargetLowering::Custom) { - ExpandShiftParts(ISD::SRL_PARTS, - Node->getOperand(0), ShiftAmt, Lo, Hi, dl); - break; - } - - // Otherwise, emit a libcall. - Lo = ExpandLibCall(RTLIB::SRL_I64, Node, false/*lshr is unsigned*/, Hi); - break; - } - - case ISD::ADD: - case ISD::SUB: { - // If the target wants to custom expand this, let them. - if (TLI.getOperationAction(Node->getOpcode(), VT) =- TargetLowering::Custom) { - SDValue Result = TLI.LowerOperation(Op, DAG); - if (Result.getNode()) { - ExpandOp(Result, Lo, Hi); - break; - } - } - // Expand the subcomponents. - SDValue LHSL, LHSH, RHSL, RHSH; - ExpandOp(Node->getOperand(0), LHSL, LHSH); - ExpandOp(Node->getOperand(1), RHSL, RHSH); - SDValue LoOps[2], HiOps[3]; - LoOps[0] = LHSL; - LoOps[1] = RHSL; - HiOps[0] = LHSH; - HiOps[1] = RHSH; - - //cascaded check to see if any smaller size has a a carry flag. - unsigned OpV = Node->getOpcode() == ISD::ADD ? ISD::ADDC : ISD::SUBC; - bool hasCarry = false; - for (unsigned BitSize = NVT.getSizeInBits(); BitSize != 0; BitSize /= 2) { - MVT AVT = MVT::getIntegerVT(BitSize); - if (TLI.isOperationLegalOrCustom(OpV, AVT)) { - hasCarry = true; - break; - } - } - - if(hasCarry) { - SDVTList VTList = DAG.getVTList(LHSL.getValueType(), MVT::Flag); - if (Node->getOpcode() == ISD::ADD) { - Lo = DAG.getNode(ISD::ADDC, dl, VTList, LoOps, 2); - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(ISD::ADDE, dl, VTList, HiOps, 3); - } else { - Lo = DAG.getNode(ISD::SUBC, dl, VTList, LoOps, 2); - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(ISD::SUBE, dl, VTList, HiOps, 3); - } - break; - } else { - if (Node->getOpcode() == ISD::ADD) { - Lo = DAG.getNode(ISD::ADD, dl, NVT, LoOps, 2); - Hi = DAG.getNode(ISD::ADD, dl, NVT, HiOps, 2); - SDValue Cmp1 = DAG.getSetCC(dl, TLI.getSetCCResultType(NVT), - Lo, LoOps[0], ISD::SETULT); - SDValue Carry1 = DAG.getNode(ISD::SELECT, dl, NVT, Cmp1, - DAG.getConstant(1, NVT), - DAG.getConstant(0, NVT)); - SDValue Cmp2 = DAG.getSetCC(dl, TLI.getSetCCResultType(NVT), - Lo, LoOps[1], ISD::SETULT); - SDValue Carry2 = DAG.getNode(ISD::SELECT, dl, NVT, Cmp2, - DAG.getConstant(1, NVT), - Carry1); - Hi = DAG.getNode(ISD::ADD, dl, NVT, Hi, Carry2); - } else { - Lo = DAG.getNode(ISD::SUB, dl, NVT, LoOps, 2); - Hi = DAG.getNode(ISD::SUB, dl, NVT, HiOps, 2); - SDValue Cmp = DAG.getSetCC(dl, NVT, LoOps[0], LoOps[1], ISD::SETULT); - SDValue Borrow = DAG.getNode(ISD::SELECT, dl, NVT, Cmp, - DAG.getConstant(1, NVT), - DAG.getConstant(0, NVT)); - Hi = DAG.getNode(ISD::SUB, dl, NVT, Hi, Borrow); - } - break; - } - } - - case ISD::ADDC: - case ISD::SUBC: { - // Expand the subcomponents. - SDValue LHSL, LHSH, RHSL, RHSH; - ExpandOp(Node->getOperand(0), LHSL, LHSH); - ExpandOp(Node->getOperand(1), RHSL, RHSH); - SDVTList VTList = DAG.getVTList(LHSL.getValueType(), MVT::Flag); - SDValue LoOps[2] = { LHSL, RHSL }; - SDValue HiOps[3] = { LHSH, RHSH }; - - if (Node->getOpcode() == ISD::ADDC) { - Lo = DAG.getNode(ISD::ADDC, dl, VTList, LoOps, 2); - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(ISD::ADDE, dl, VTList, HiOps, 3); - } else { - Lo = DAG.getNode(ISD::SUBC, dl, VTList, LoOps, 2); - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(ISD::SUBE, dl, VTList, HiOps, 3); - } - // Remember that we legalized the flag. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Hi.getValue(1))); - break; - } - case ISD::ADDE: - case ISD::SUBE: { - // Expand the subcomponents. - SDValue LHSL, LHSH, RHSL, RHSH; - ExpandOp(Node->getOperand(0), LHSL, LHSH); - ExpandOp(Node->getOperand(1), RHSL, RHSH); - SDVTList VTList = DAG.getVTList(LHSL.getValueType(), MVT::Flag); - SDValue LoOps[3] = { LHSL, RHSL, Node->getOperand(2) }; - SDValue HiOps[3] = { LHSH, RHSH }; - - Lo = DAG.getNode(Node->getOpcode(), dl, VTList, LoOps, 3); - HiOps[2] = Lo.getValue(1); - Hi = DAG.getNode(Node->getOpcode(), dl, VTList, HiOps, 3); - - // Remember that we legalized the flag. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Hi.getValue(1))); - break; - } - case ISD::MUL: { - // If the target wants to custom expand this, let them. - if (TLI.getOperationAction(ISD::MUL, VT) == TargetLowering::Custom) { - SDValue New = TLI.LowerOperation(Op, DAG); - if (New.getNode()) { - ExpandOp(New, Lo, Hi); - break; - } - } - - bool HasMULHS = TLI.isOperationLegalOrCustom(ISD::MULHS, NVT); - bool HasMULHU = TLI.isOperationLegalOrCustom(ISD::MULHU, NVT); - bool HasSMUL_LOHI = TLI.isOperationLegalOrCustom(ISD::SMUL_LOHI, NVT); - bool HasUMUL_LOHI = TLI.isOperationLegalOrCustom(ISD::UMUL_LOHI, NVT); - if (HasMULHU || HasMULHS || HasUMUL_LOHI || HasSMUL_LOHI) { - SDValue LL, LH, RL, RH; - ExpandOp(Node->getOperand(0), LL, LH); - ExpandOp(Node->getOperand(1), RL, RH); - unsigned OuterBitSize = Op.getValueSizeInBits(); - unsigned InnerBitSize = RH.getValueSizeInBits(); - unsigned LHSSB = DAG.ComputeNumSignBits(Op.getOperand(0)); - unsigned RHSSB = DAG.ComputeNumSignBits(Op.getOperand(1)); - APInt HighMask = APInt::getHighBitsSet(OuterBitSize, InnerBitSize); - if (DAG.MaskedValueIsZero(Node->getOperand(0), HighMask) && - DAG.MaskedValueIsZero(Node->getOperand(1), HighMask)) { - // The inputs are both zero-extended. - if (HasUMUL_LOHI) { - // We can emit a umul_lohi. - Lo = DAG.getNode(ISD::UMUL_LOHI, dl, DAG.getVTList(NVT, NVT), LL, RL); - Hi = SDValue(Lo.getNode(), 1); - break; - } - if (HasMULHU) { - // We can emit a mulhu+mul. - Lo = DAG.getNode(ISD::MUL, dl, NVT, LL, RL); - Hi = DAG.getNode(ISD::MULHU, dl, NVT, LL, RL); - break; - } - } - if (LHSSB > InnerBitSize && RHSSB > InnerBitSize) { - // The input values are both sign-extended. - if (HasSMUL_LOHI) { - // We can emit a smul_lohi. - Lo = DAG.getNode(ISD::SMUL_LOHI, dl, DAG.getVTList(NVT, NVT), LL, RL); - Hi = SDValue(Lo.getNode(), 1); - break; - } - if (HasMULHS) { - // We can emit a mulhs+mul. - Lo = DAG.getNode(ISD::MUL, dl, NVT, LL, RL); - Hi = DAG.getNode(ISD::MULHS, dl, NVT, LL, RL); - break; - } - } - if (HasUMUL_LOHI) { - // Lo,Hi = umul LHS, RHS. - SDValue UMulLOHI = DAG.getNode(ISD::UMUL_LOHI, dl, - DAG.getVTList(NVT, NVT), LL, RL); - Lo = UMulLOHI; - Hi = UMulLOHI.getValue(1); - RH = DAG.getNode(ISD::MUL, dl, NVT, LL, RH); - LH = DAG.getNode(ISD::MUL, dl, NVT, LH, RL); - Hi = DAG.getNode(ISD::ADD, dl, NVT, Hi, RH); - Hi = DAG.getNode(ISD::ADD, dl, NVT, Hi, LH); - break; - } - if (HasMULHU) { - Lo = DAG.getNode(ISD::MUL, dl, NVT, LL, RL); - Hi = DAG.getNode(ISD::MULHU, dl, NVT, LL, RL); - RH = DAG.getNode(ISD::MUL, dl, NVT, LL, RH); - LH = DAG.getNode(ISD::MUL, dl, NVT, LH, RL); - Hi = DAG.getNode(ISD::ADD, dl, NVT, Hi, RH); - Hi = DAG.getNode(ISD::ADD, dl, NVT, Hi, LH); - break; - } - } - - // If nothing else, we can make a libcall. - Lo = ExpandLibCall(RTLIB::MUL_I64, Node, false/*sign irrelevant*/, Hi); - break; - } - case ISD::SDIV: - Lo = ExpandLibCall(RTLIB::SDIV_I64, Node, true, Hi); - break; - case ISD::UDIV: - Lo = ExpandLibCall(RTLIB::UDIV_I64, Node, true, Hi); - break; - case ISD::SREM: - Lo = ExpandLibCall(RTLIB::SREM_I64, Node, true, Hi); - break; - case ISD::UREM: - Lo = ExpandLibCall(RTLIB::UREM_I64, Node, true, Hi); - break; - - case ISD::FADD: - Lo = ExpandLibCall(GetFPLibCall(VT, RTLIB::ADD_F32, - RTLIB::ADD_F64, - RTLIB::ADD_F80, - RTLIB::ADD_PPCF128), - Node, false, Hi); - break; - case ISD::FSUB: - Lo = ExpandLibCall(GetFPLibCall(VT, RTLIB::SUB_F32, - RTLIB::SUB_F64, - RTLIB::SUB_F80, - RTLIB::SUB_PPCF128), - Node, false, Hi); - break; - case ISD::FMUL: - Lo = ExpandLibCall(GetFPLibCall(VT, RTLIB::MUL_F32, - RTLIB::MUL_F64, - RTLIB::MUL_F80, - RTLIB::MUL_PPCF128), - Node, false, Hi); - break; - case ISD::FDIV: - Lo = ExpandLibCall(GetFPLibCall(VT, RTLIB::DIV_F32, - RTLIB::DIV_F64, - RTLIB::DIV_F80, - RTLIB::DIV_PPCF128), - Node, false, Hi); - break; - case ISD::FP_EXTEND: { - if (VT == MVT::ppcf128) { - assert(Node->getOperand(0).getValueType()==MVT::f32 || - Node->getOperand(0).getValueType()==MVT::f64); - const uint64_t zero = 0; - if (Node->getOperand(0).getValueType()==MVT::f32) - Hi = DAG.getNode(ISD::FP_EXTEND, dl, MVT::f64, Node->getOperand(0)); - else - Hi = Node->getOperand(0); - Lo = DAG.getConstantFP(APFloat(APInt(64, 1, &zero)), MVT::f64); - break; - } - RTLIB::Libcall LC = RTLIB::getFPEXT(Node->getOperand(0).getValueType(), VT); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unsupported FP_EXTEND!"); - Lo = ExpandLibCall(LC, Node, true, Hi); - break; - } - case ISD::FP_ROUND: { - RTLIB::Libcall LC = RTLIB::getFPROUND(Node->getOperand(0).getValueType(), - VT); - assert(LC != RTLIB::UNKNOWN_LIBCALL && "Unsupported FP_ROUND!"); - Lo = ExpandLibCall(LC, Node, true, Hi); - break; - } - case ISD::FSQRT: - case ISD::FSIN: - case ISD::FCOS: - case ISD::FLOG: - case ISD::FLOG2: - case ISD::FLOG10: - case ISD::FEXP: - case ISD::FEXP2: - case ISD::FTRUNC: - case ISD::FFLOOR: - case ISD::FCEIL: - case ISD::FRINT: - case ISD::FNEARBYINT: - case ISD::FPOW: - case ISD::FPOWI: { - RTLIB::Libcall LC = RTLIB::UNKNOWN_LIBCALL; - switch(Node->getOpcode()) { - case ISD::FSQRT: - LC = GetFPLibCall(VT, RTLIB::SQRT_F32, RTLIB::SQRT_F64, - RTLIB::SQRT_F80, RTLIB::SQRT_PPCF128); - break; - case ISD::FSIN: - LC = GetFPLibCall(VT, RTLIB::SIN_F32, RTLIB::SIN_F64, - RTLIB::SIN_F80, RTLIB::SIN_PPCF128); - break; - case ISD::FCOS: - LC = GetFPLibCall(VT, RTLIB::COS_F32, RTLIB::COS_F64, - RTLIB::COS_F80, RTLIB::COS_PPCF128); - break; - case ISD::FLOG: - LC = GetFPLibCall(VT, RTLIB::LOG_F32, RTLIB::LOG_F64, - RTLIB::LOG_F80, RTLIB::LOG_PPCF128); - break; - case ISD::FLOG2: - LC = GetFPLibCall(VT, RTLIB::LOG2_F32, RTLIB::LOG2_F64, - RTLIB::LOG2_F80, RTLIB::LOG2_PPCF128); - break; - case ISD::FLOG10: - LC = GetFPLibCall(VT, RTLIB::LOG10_F32, RTLIB::LOG10_F64, - RTLIB::LOG10_F80, RTLIB::LOG10_PPCF128); - break; - case ISD::FEXP: - LC = GetFPLibCall(VT, RTLIB::EXP_F32, RTLIB::EXP_F64, - RTLIB::EXP_F80, RTLIB::EXP_PPCF128); - break; - case ISD::FEXP2: - LC = GetFPLibCall(VT, RTLIB::EXP2_F32, RTLIB::EXP2_F64, - RTLIB::EXP2_F80, RTLIB::EXP2_PPCF128); - break; - case ISD::FTRUNC: - LC = GetFPLibCall(VT, RTLIB::TRUNC_F32, RTLIB::TRUNC_F64, - RTLIB::TRUNC_F80, RTLIB::TRUNC_PPCF128); - break; - case ISD::FFLOOR: - LC = GetFPLibCall(VT, RTLIB::FLOOR_F32, RTLIB::FLOOR_F64, - RTLIB::FLOOR_F80, RTLIB::FLOOR_PPCF128); - break; - case ISD::FCEIL: - LC = GetFPLibCall(VT, RTLIB::CEIL_F32, RTLIB::CEIL_F64, - RTLIB::CEIL_F80, RTLIB::CEIL_PPCF128); - break; - case ISD::FRINT: - LC = GetFPLibCall(VT, RTLIB::RINT_F32, RTLIB::RINT_F64, - RTLIB::RINT_F80, RTLIB::RINT_PPCF128); - break; - case ISD::FNEARBYINT: - LC = GetFPLibCall(VT, RTLIB::NEARBYINT_F32, RTLIB::NEARBYINT_F64, - RTLIB::NEARBYINT_F80, RTLIB::NEARBYINT_PPCF128); - break; - case ISD::FPOW: - LC = GetFPLibCall(VT, RTLIB::POW_F32, RTLIB::POW_F64, RTLIB::POW_F80, - RTLIB::POW_PPCF128); - break; - case ISD::FPOWI: - LC = GetFPLibCall(VT, RTLIB::POWI_F32, RTLIB::POWI_F64, RTLIB::POWI_F80, - RTLIB::POWI_PPCF128); - break; - default: assert(0 && "Unreachable!"); - } - Lo = ExpandLibCall(LC, Node, false, Hi); - break; - } - case ISD::FABS: { - if (VT == MVT::ppcf128) { - SDValue Tmp; - ExpandOp(Node->getOperand(0), Lo, Tmp); - Hi = DAG.getNode(ISD::FABS, dl, NVT, Tmp); - // lo = hi==fabs(hi) ? lo : -lo; - Lo = DAG.getNode(ISD::SELECT_CC, dl, NVT, Hi, Tmp, - Lo, DAG.getNode(ISD::FNEG, dl, NVT, Lo), - DAG.getCondCode(ISD::SETEQ)); - break; - } - SDValue Mask = (VT == MVT::f64) - ? DAG.getConstantFP(BitsToDouble(~(1ULL << 63)), VT) - : DAG.getConstantFP(BitsToFloat(~(1U << 31)), VT); - Mask = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Mask); - Lo = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Node->getOperand(0)); - Lo = DAG.getNode(ISD::AND, dl, NVT, Lo, Mask); - if (getTypeAction(NVT) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - case ISD::FNEG: { - if (VT == MVT::ppcf128) { - ExpandOp(Node->getOperand(0), Lo, Hi); - Lo = DAG.getNode(ISD::FNEG, dl, MVT::f64, Lo); - Hi = DAG.getNode(ISD::FNEG, dl, MVT::f64, Hi); - break; - } - SDValue Mask = (VT == MVT::f64) - ? DAG.getConstantFP(BitsToDouble(1ULL << 63), VT) - : DAG.getConstantFP(BitsToFloat(1U << 31), VT); - Mask = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Mask); - Lo = DAG.getNode(ISD::BIT_CONVERT, dl, NVT, Node->getOperand(0)); - Lo = DAG.getNode(ISD::XOR, dl, NVT, Lo, Mask); - if (getTypeAction(NVT) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - case ISD::FCOPYSIGN: { - Lo = ExpandFCOPYSIGNToBitwiseOps(Node, NVT, DAG, TLI); - if (getTypeAction(NVT) == Expand) - ExpandOp(Lo, Lo, Hi); - break; - } - case ISD::SINT_TO_FP: - case ISD::UINT_TO_FP: { - bool isSigned = Node->getOpcode() == ISD::SINT_TO_FP; - MVT SrcVT = Node->getOperand(0).getValueType(); - - // Promote the operand if needed. Do this before checking for - // ppcf128 so conversions of i16 and i8 work. - if (getTypeAction(SrcVT) == Promote) { - SDValue Tmp = PromoteOp(Node->getOperand(0)); - Tmp = isSigned - ? DAG.getNode(ISD::SIGN_EXTEND_INREG, dl, Tmp.getValueType(), Tmp, - DAG.getValueType(SrcVT)) - : DAG.getZeroExtendInReg(Tmp, dl, SrcVT); - Node = DAG.UpdateNodeOperands(Op, Tmp).getNode(); - SrcVT = Node->getOperand(0).getValueType(); - } - - if (VT == MVT::ppcf128 && SrcVT == MVT::i32) { - static const uint64_t zero = 0; - if (isSigned) { - Hi = LegalizeOp(DAG.getNode(ISD::SINT_TO_FP, dl, MVT::f64, - Node->getOperand(0))); - Lo = DAG.getConstantFP(APFloat(APInt(64, 1, &zero)), MVT::f64); - } else { - static const uint64_t TwoE32[] = { 0x41f0000000000000LL, 0 }; - Hi = LegalizeOp(DAG.getNode(ISD::SINT_TO_FP, dl, MVT::f64, - Node->getOperand(0))); - Lo = DAG.getConstantFP(APFloat(APInt(64, 1, &zero)), MVT::f64); - Hi = DAG.getNode(ISD::BUILD_PAIR, dl, VT, Lo, Hi); - // X>=0 ? {(f64)x, 0} : {(f64)x, 0} + 2^32 - ExpandOp(DAG.getNode(ISD::SELECT_CC, dl, - MVT::ppcf128, Node->getOperand(0), - DAG.getConstant(0, MVT::i32), - DAG.getNode(ISD::FADD, dl, MVT::ppcf128, Hi, - DAG.getConstantFP - (APFloat(APInt(128, 2, TwoE32)), - MVT::ppcf128)), - Hi, - DAG.getCondCode(ISD::SETLT)), - Lo, Hi); - } - break; - } - if (VT == MVT::ppcf128 && SrcVT == MVT::i64 && !isSigned) { - // si64->ppcf128 done by libcall, below - static const uint64_t TwoE64[] = { 0x43f0000000000000LL, 0 }; - ExpandOp(DAG.getNode(ISD::SINT_TO_FP, dl, MVT::ppcf128, - Node->getOperand(0)), Lo, Hi); - Hi = DAG.getNode(ISD::BUILD_PAIR, dl, VT, Lo, Hi); - // x>=0 ? (ppcf128)(i64)x : (ppcf128)(i64)x + 2^64 - ExpandOp(DAG.getNode(ISD::SELECT_CC, dl, MVT::ppcf128, - Node->getOperand(0), - DAG.getConstant(0, MVT::i64), - DAG.getNode(ISD::FADD, dl, MVT::ppcf128, Hi, - DAG.getConstantFP - (APFloat(APInt(128, 2, TwoE64)), - MVT::ppcf128)), - Hi, - DAG.getCondCode(ISD::SETLT)), - Lo, Hi); - break; - } - - Lo = ExpandIntToFP(Node->getOpcode() == ISD::SINT_TO_FP, VT, - Node->getOperand(0), dl); - if (getTypeAction(Lo.getValueType()) == Expand) - // float to i32 etc. can be 'expanded' to a single node. - ExpandOp(Lo, Lo, Hi); - break; - } - } - - // Make sure the resultant values have been legalized themselves, unless this - // is a type that requires multi-step expansion. - if (getTypeAction(NVT) != Expand && NVT != MVT::isVoid) { - Lo = LegalizeOp(Lo); - if (Hi.getNode()) - // Don't legalize the high part if it is expanded to a single node. - Hi = LegalizeOp(Hi); - } - - // Remember in a map if the values will be reused later. - bool isNew - ExpandedNodes.insert(std::make_pair(Op, std::make_pair(Lo, Hi))).second; - assert(isNew && "Value already expanded?!?"); - isNew = isNew; -} - -/// SplitVectorOp - Given an operand of vector type, break it down into -/// two smaller values, still of vector type. -void SelectionDAGLegalize::SplitVectorOp(SDValue Op, SDValue &Lo, - SDValue &Hi) { - assert(Op.getValueType().isVector() && "Cannot split non-vector type!"); - SDNode *Node = Op.getNode(); - DebugLoc dl = Node->getDebugLoc(); - unsigned NumElements = Op.getValueType().getVectorNumElements(); - assert(NumElements > 1 && "Cannot split a single element vector!"); - - MVT NewEltVT = Op.getValueType().getVectorElementType(); - - unsigned NewNumElts_Lo = 1 << Log2_32(NumElements-1); - unsigned NewNumElts_Hi = NumElements - NewNumElts_Lo; - - MVT NewVT_Lo = MVT::getVectorVT(NewEltVT, NewNumElts_Lo); - MVT NewVT_Hi = MVT::getVectorVT(NewEltVT, NewNumElts_Hi); - - // See if we already split it. - std::map<SDValue, std::pair<SDValue, SDValue> >::iterator I - = SplitNodes.find(Op); - if (I != SplitNodes.end()) { - Lo = I->second.first; - Hi = I->second.second; - return; - } - - switch (Node->getOpcode()) { - default: -#ifndef NDEBUG - Node->dump(&DAG); -#endif - assert(0 && "Unhandled operation in SplitVectorOp!"); - case ISD::UNDEF: - Lo = DAG.getUNDEF(NewVT_Lo); - Hi = DAG.getUNDEF(NewVT_Hi); - break; - case ISD::BUILD_PAIR: - Lo = Node->getOperand(0); - Hi = Node->getOperand(1); - break; - case ISD::INSERT_VECTOR_ELT: { - if (ConstantSDNode *Idx = dyn_cast<ConstantSDNode>(Node->getOperand(2))) { - SplitVectorOp(Node->getOperand(0), Lo, Hi); - unsigned Index = Idx->getZExtValue(); - SDValue ScalarOp = Node->getOperand(1); - if (Index < NewNumElts_Lo) - Lo = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, NewVT_Lo, Lo, ScalarOp, - DAG.getIntPtrConstant(Index)); - else - Hi = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, NewVT_Hi, Hi, ScalarOp, - DAG.getIntPtrConstant(Index - NewNumElts_Lo)); - break; - } - SDValue Tmp = PerformInsertVectorEltInMemory(Node->getOperand(0), - Node->getOperand(1), - Node->getOperand(2), dl); - SplitVectorOp(Tmp, Lo, Hi); - break; - } - case ISD::VECTOR_SHUFFLE: { - // Build the low part. - SDValue Mask = Node->getOperand(2); - SmallVector<SDValue, 8> Ops; - MVT PtrVT = TLI.getPointerTy(); - - // Insert all of the elements from the input that are needed. We use - // buildvector of extractelement here because the input vectors will have - // to be legalized, so this makes the code simpler. - for (unsigned i = 0; i != NewNumElts_Lo; ++i) { - SDValue IdxNode = Mask.getOperand(i); - if (IdxNode.getOpcode() == ISD::UNDEF) { - Ops.push_back(DAG.getUNDEF(NewEltVT)); - continue; - } - unsigned Idx = cast<ConstantSDNode>(IdxNode)->getZExtValue(); - SDValue InVec = Node->getOperand(0); - if (Idx >= NumElements) { - InVec = Node->getOperand(1); - Idx -= NumElements; - } - Ops.push_back(DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, NewEltVT, InVec, - DAG.getConstant(Idx, PtrVT))); - } - Lo = DAG.getNode(ISD::BUILD_VECTOR, dl, NewVT_Lo, &Ops[0], Ops.size()); - Ops.clear(); - - for (unsigned i = NewNumElts_Lo; i != NumElements; ++i) { - SDValue IdxNode = Mask.getOperand(i); - if (IdxNode.getOpcode() == ISD::UNDEF) { - Ops.push_back(DAG.getUNDEF(NewEltVT)); - continue; - } - unsigned Idx = cast<ConstantSDNode>(IdxNode)->getZExtValue(); - SDValue InVec = Node->getOperand(0); - if (Idx >= NumElements) { - InVec = Node->getOperand(1); - Idx -= NumElements; - } - Ops.push_back(DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, NewEltVT, InVec, - DAG.getConstant(Idx, PtrVT))); - } - Hi = DAG.getNode(ISD::BUILD_VECTOR, dl, NewVT_Hi, &Ops[0], Ops.size()); - break; - } - case ISD::BUILD_VECTOR: { - SmallVector<SDValue, 8> LoOps(Node->op_begin(), - Node->op_begin()+NewNumElts_Lo); - Lo = DAG.getNode(ISD::BUILD_VECTOR, dl, NewVT_Lo, &LoOps[0], LoOps.size()); - - SmallVector<SDValue, 8> HiOps(Node->op_begin()+NewNumElts_Lo, - Node->op_end()); - Hi = DAG.getNode(ISD::BUILD_VECTOR, dl, NewVT_Hi, &HiOps[0], HiOps.size()); - break; - } - case ISD::CONCAT_VECTORS: { - // FIXME: Handle non-power-of-two vectors? - unsigned NewNumSubvectors = Node->getNumOperands() / 2; - if (NewNumSubvectors == 1) { - Lo = Node->getOperand(0); - Hi = Node->getOperand(1); - } else { - SmallVector<SDValue, 8> LoOps(Node->op_begin(), - Node->op_begin()+NewNumSubvectors); - Lo = DAG.getNode(ISD::CONCAT_VECTORS, dl, NewVT_Lo, - &LoOps[0], LoOps.size()); - - SmallVector<SDValue, 8> HiOps(Node->op_begin()+NewNumSubvectors, - Node->op_end()); - Hi = DAG.getNode(ISD::CONCAT_VECTORS, dl, NewVT_Hi, - &HiOps[0], HiOps.size()); - } - break; - } - case ISD::EXTRACT_SUBVECTOR: { - SDValue Vec = Op.getOperand(0); - SDValue Idx = Op.getOperand(1); - MVT IdxVT = Idx.getValueType(); - - Lo = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, NewVT_Lo, Vec, Idx); - ConstantSDNode *CIdx = dyn_cast<ConstantSDNode>(Idx); - if (CIdx) { - Hi = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, NewVT_Hi, Vec, - DAG.getConstant(CIdx->getZExtValue() + NewNumElts_Lo, - IdxVT)); - } else { - Idx = DAG.getNode(ISD::ADD, dl, IdxVT, Idx, - DAG.getConstant(NewNumElts_Lo, IdxVT)); - Hi = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, NewVT_Hi, Vec, Idx); - } - break; - } - case ISD::SELECT: { - SDValue Cond = Node->getOperand(0); - - SDValue LL, LH, RL, RH; - SplitVectorOp(Node->getOperand(1), LL, LH); - SplitVectorOp(Node->getOperand(2), RL, RH); - - if (Cond.getValueType().isVector()) { - // Handle a vector merge. - SDValue CL, CH; - SplitVectorOp(Cond, CL, CH); - Lo = DAG.getNode(Node->getOpcode(), dl, NewVT_Lo, CL, LL, RL); - Hi = DAG.getNode(Node->getOpcode(), dl, NewVT_Hi, CH, LH, RH); - } else { - // Handle a simple select with vector operands. - Lo = DAG.getNode(Node->getOpcode(), dl, NewVT_Lo, Cond, LL, RL); - Hi = DAG.getNode(Node->getOpcode(), dl, NewVT_Hi, Cond, LH, RH); - } - break; - } - case ISD::SELECT_CC: { - SDValue CondLHS = Node->getOperand(0); - SDValue CondRHS = Node->getOperand(1); - SDValue CondCode = Node->getOperand(4); - - SDValue LL, LH, RL, RH; - SplitVectorOp(Node->getOperand(2), LL, LH); - SplitVectorOp(Node->getOperand(3), RL, RH); - - // Handle a simple select with vector operands. - Lo = DAG.getNode(ISD::SELECT_CC, dl, NewVT_Lo, CondLHS, CondRHS, - LL, RL, CondCode); - Hi = DAG.getNode(ISD::SELECT_CC, dl, NewVT_Hi, CondLHS, CondRHS, - LH, RH, CondCode); - break; - } - case ISD::VSETCC: { - SDValue LL, LH, RL, RH; - SplitVectorOp(Node->getOperand(0), LL, LH); - SplitVectorOp(Node->getOperand(1), RL, RH); - Lo = DAG.getNode(ISD::VSETCC, dl, NewVT_Lo, LL, RL, Node->getOperand(2)); - Hi = DAG.getNode(ISD::VSETCC, dl, NewVT_Hi, LH, RH, Node->getOperand(2)); - break; - } - case ISD::ADD: - case ISD::SUB: - case ISD::MUL: - case ISD::FADD: - case ISD::FSUB: - case ISD::FMUL: - case ISD::SDIV: - case ISD::UDIV: - case ISD::FDIV: - case ISD::FPOW: - case ISD::AND: - case ISD::OR: - case ISD::XOR: - case ISD::UREM: - case ISD::SREM: - case ISD::FREM: - case ISD::SHL: - case ISD::SRA: - case ISD::SRL: { - SDValue LL, LH, RL, RH; - SplitVectorOp(Node->getOperand(0), LL, LH); - SplitVectorOp(Node->getOperand(1), RL, RH); - - Lo = DAG.getNode(Node->getOpcode(), dl, NewVT_Lo, LL, RL); - Hi = DAG.getNode(Node->getOpcode(), dl, NewVT_Hi, LH, RH); - break; - } - case ISD::FP_ROUND: - case ISD::FPOWI: { - SDValue L, H; - SplitVectorOp(Node->getOperand(0), L, H); - - Lo = DAG.getNode(Node->getOpcode(), dl, NewVT_Lo, L, Node->getOperand(1)); - Hi = DAG.getNode(Node->getOpcode(), dl, NewVT_Hi, H, Node->getOperand(1)); - break; - } - case ISD::CTTZ: - case ISD::CTLZ: - case ISD::CTPOP: - case ISD::FNEG: - case ISD::FABS: - case ISD::FSQRT: - case ISD::FSIN: - case ISD::FCOS: - case ISD::FLOG: - case ISD::FLOG2: - case ISD::FLOG10: - case ISD::FEXP: - case ISD::FEXP2: - case ISD::FP_TO_SINT: - case ISD::FP_TO_UINT: - case ISD::SINT_TO_FP: - case ISD::UINT_TO_FP: - case ISD::TRUNCATE: - case ISD::ANY_EXTEND: - case ISD::SIGN_EXTEND: - case ISD::ZERO_EXTEND: - case ISD::FP_EXTEND: { - SDValue L, H; - SplitVectorOp(Node->getOperand(0), L, H); - - Lo = DAG.getNode(Node->getOpcode(), dl, NewVT_Lo, L); - Hi = DAG.getNode(Node->getOpcode(), dl, NewVT_Hi, H); - break; - } - case ISD::CONVERT_RNDSAT: { - ISD::CvtCode CvtCode = cast<CvtRndSatSDNode>(Node)->getCvtCode(); - SDValue L, H; - SplitVectorOp(Node->getOperand(0), L, H); - SDValue DTyOpL = DAG.getValueType(NewVT_Lo); - SDValue DTyOpH = DAG.getValueType(NewVT_Hi); - SDValue STyOpL = DAG.getValueType(L.getValueType()); - SDValue STyOpH = DAG.getValueType(H.getValueType()); - - SDValue RndOp = Node->getOperand(3); - SDValue SatOp = Node->getOperand(4); - - Lo = DAG.getConvertRndSat(NewVT_Lo, dl, L, DTyOpL, STyOpL, - RndOp, SatOp, CvtCode); - Hi = DAG.getConvertRndSat(NewVT_Hi, dl, H, DTyOpH, STyOpH, - RndOp, SatOp, CvtCode); - break; - } - case ISD::LOAD: { - LoadSDNode *LD = cast<LoadSDNode>(Node); - SDValue Ch = LD->getChain(); - SDValue Ptr = LD->getBasePtr(); - ISD::LoadExtType ExtType = LD->getExtensionType(); - const Value *SV = LD->getSrcValue(); - int SVOffset = LD->getSrcValueOffset(); - MVT MemoryVT = LD->getMemoryVT(); - unsigned Alignment = LD->getAlignment(); - bool isVolatile = LD->isVolatile(); - - assert(LD->isUnindexed() && "Indexed vector loads are not supported yet!"); - SDValue Offset = DAG.getUNDEF(Ptr.getValueType()); - - MVT MemNewEltVT = MemoryVT.getVectorElementType(); - MVT MemNewVT_Lo = MVT::getVectorVT(MemNewEltVT, NewNumElts_Lo); - MVT MemNewVT_Hi = MVT::getVectorVT(MemNewEltVT, NewNumElts_Hi); - - Lo = DAG.getLoad(ISD::UNINDEXED, dl, ExtType, - NewVT_Lo, Ch, Ptr, Offset, - SV, SVOffset, MemNewVT_Lo, isVolatile, Alignment); - unsigned IncrementSize = NewNumElts_Lo * MemNewEltVT.getSizeInBits()/8; - Ptr = DAG.getNode(ISD::ADD, dl, Ptr.getValueType(), Ptr, - DAG.getIntPtrConstant(IncrementSize)); - SVOffset += IncrementSize; - Alignment = MinAlign(Alignment, IncrementSize); - Hi = DAG.getLoad(ISD::UNINDEXED, dl, ExtType, - NewVT_Hi, Ch, Ptr, Offset, - SV, SVOffset, MemNewVT_Hi, isVolatile, Alignment); - - // Build a factor node to remember that this load is independent of the - // other one. - SDValue TF = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, Lo.getValue(1), - Hi.getValue(1)); - - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(TF)); - break; - } - case ISD::BIT_CONVERT: { - // We know the result is a vector. The input may be either a vector or a - // scalar value. - SDValue InOp = Node->getOperand(0); - if (!InOp.getValueType().isVector() || - InOp.getValueType().getVectorNumElements() == 1) { - // The input is a scalar or single-element vector. - // Lower to a store/load so that it can be split. - // FIXME: this could be improved probably. - unsigned LdAlign = TLI.getTargetData()-> - getPrefTypeAlignment(Op.getValueType().getTypeForMVT()); - SDValue Ptr = DAG.CreateStackTemporary(InOp.getValueType(), LdAlign); - int FI = cast<FrameIndexSDNode>(Ptr.getNode())->getIndex(); - - SDValue St = DAG.getStore(DAG.getEntryNode(), dl, - InOp, Ptr, - PseudoSourceValue::getFixedStack(FI), 0); - InOp = DAG.getLoad(Op.getValueType(), dl, St, Ptr, - PseudoSourceValue::getFixedStack(FI), 0); - } - // Split the vector and convert each of the pieces now. - SplitVectorOp(InOp, Lo, Hi); - Lo = DAG.getNode(ISD::BIT_CONVERT, dl, NewVT_Lo, Lo); - Hi = DAG.getNode(ISD::BIT_CONVERT, dl, NewVT_Hi, Hi); - break; - } - } - - // Remember in a map if the values will be reused later. - bool isNew - SplitNodes.insert(std::make_pair(Op, std::make_pair(Lo, Hi))).second; - assert(isNew && "Value already split?!?"); - isNew = isNew; -} - - -/// ScalarizeVectorOp - Given an operand of single-element vector type -/// (e.g. v1f32), convert it into the equivalent operation that returns a -/// scalar (e.g. f32) value. -SDValue SelectionDAGLegalize::ScalarizeVectorOp(SDValue Op) { - assert(Op.getValueType().isVector() && "Bad ScalarizeVectorOp invocation!"); - SDNode *Node = Op.getNode(); - DebugLoc dl = Node->getDebugLoc(); - MVT NewVT = Op.getValueType().getVectorElementType(); - assert(Op.getValueType().getVectorNumElements() == 1); - - // See if we already scalarized it. - std::map<SDValue, SDValue>::iterator I = ScalarizedNodes.find(Op); - if (I != ScalarizedNodes.end()) return I->second; - - SDValue Result; - switch (Node->getOpcode()) { - default: -#ifndef NDEBUG - Node->dump(&DAG); cerr << "\n"; -#endif - assert(0 && "Unknown vector operation in ScalarizeVectorOp!"); - case ISD::ADD: - case ISD::FADD: - case ISD::SUB: - case ISD::FSUB: - case ISD::MUL: - case ISD::FMUL: - case ISD::SDIV: - case ISD::UDIV: - case ISD::FDIV: - case ISD::SREM: - case ISD::UREM: - case ISD::FREM: - case ISD::FPOW: - case ISD::AND: - case ISD::OR: - case ISD::XOR: - Result = DAG.getNode(Node->getOpcode(), dl, - NewVT, - ScalarizeVectorOp(Node->getOperand(0)), - ScalarizeVectorOp(Node->getOperand(1))); - break; - case ISD::FNEG: - case ISD::FABS: - case ISD::FSQRT: - case ISD::FSIN: - case ISD::FCOS: - case ISD::FLOG: - case ISD::FLOG2: - case ISD::FLOG10: - case ISD::FEXP: - case ISD::FEXP2: - case ISD::FP_TO_SINT: - case ISD::FP_TO_UINT: - case ISD::SINT_TO_FP: - case ISD::UINT_TO_FP: - case ISD::SIGN_EXTEND: - case ISD::ZERO_EXTEND: - case ISD::ANY_EXTEND: - case ISD::TRUNCATE: - case ISD::FP_EXTEND: - Result = DAG.getNode(Node->getOpcode(), dl, - NewVT, - ScalarizeVectorOp(Node->getOperand(0))); - break; - case ISD::CONVERT_RNDSAT: { - SDValue Op0 = ScalarizeVectorOp(Node->getOperand(0)); - Result = DAG.getConvertRndSat(NewVT, dl, Op0, - DAG.getValueType(NewVT), - DAG.getValueType(Op0.getValueType()), - Node->getOperand(3), - Node->getOperand(4), - cast<CvtRndSatSDNode>(Node)->getCvtCode()); - break; - } - case ISD::FPOWI: - case ISD::FP_ROUND: - Result = DAG.getNode(Node->getOpcode(), dl, - NewVT, - ScalarizeVectorOp(Node->getOperand(0)), - Node->getOperand(1)); - break; - case ISD::LOAD: { - LoadSDNode *LD = cast<LoadSDNode>(Node); - SDValue Ch = LegalizeOp(LD->getChain()); // Legalize the chain. - SDValue Ptr = LegalizeOp(LD->getBasePtr()); // Legalize the pointer. - ISD::LoadExtType ExtType = LD->getExtensionType(); - const Value *SV = LD->getSrcValue(); - int SVOffset = LD->getSrcValueOffset(); - MVT MemoryVT = LD->getMemoryVT(); - unsigned Alignment = LD->getAlignment(); - bool isVolatile = LD->isVolatile(); - - assert(LD->isUnindexed() && "Indexed vector loads are not supported yet!"); - SDValue Offset = DAG.getUNDEF(Ptr.getValueType()); - - Result = DAG.getLoad(ISD::UNINDEXED, dl, ExtType, - NewVT, Ch, Ptr, Offset, SV, SVOffset, - MemoryVT.getVectorElementType(), - isVolatile, Alignment); - - // Remember that we legalized the chain. - AddLegalizedOperand(Op.getValue(1), LegalizeOp(Result.getValue(1))); - break; - } - case ISD::BUILD_VECTOR: - Result = Node->getOperand(0); - break; - case ISD::INSERT_VECTOR_ELT: - // Returning the inserted scalar element. - Result = Node->getOperand(1); - break; - case ISD::CONCAT_VECTORS: - assert(Node->getOperand(0).getValueType() == NewVT && - "Concat of non-legal vectors not yet supported!"); - Result = Node->getOperand(0); - break; - case ISD::VECTOR_SHUFFLE: { - // Figure out if the scalar is the LHS or RHS and return it. - SDValue EltNum = Node->getOperand(2).getOperand(0); - if (cast<ConstantSDNode>(EltNum)->getZExtValue()) - Result = ScalarizeVectorOp(Node->getOperand(1)); - else - Result = ScalarizeVectorOp(Node->getOperand(0)); - break; - } - case ISD::EXTRACT_SUBVECTOR: - Result = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, NewVT, - Node->getOperand(0), Node->getOperand(1)); - break; - case ISD::BIT_CONVERT: { - SDValue Op0 = Op.getOperand(0); - if (Op0.getValueType().isVector() && - Op0.getValueType().getVectorNumElements() == 1) - Op0 = ScalarizeVectorOp(Op0); - Result = DAG.getNode(ISD::BIT_CONVERT, dl, NewVT, Op0); - break; - } - case ISD::SELECT: - Result = DAG.getNode(ISD::SELECT, dl, NewVT, Op.getOperand(0), - ScalarizeVectorOp(Op.getOperand(1)), - ScalarizeVectorOp(Op.getOperand(2))); - break; - case ISD::SELECT_CC: - Result = DAG.getNode(ISD::SELECT_CC, dl, NewVT, Node->getOperand(0), - Node->getOperand(1), - ScalarizeVectorOp(Op.getOperand(2)), - ScalarizeVectorOp(Op.getOperand(3)), - Node->getOperand(4)); - break; - case ISD::VSETCC: { - SDValue Op0 = ScalarizeVectorOp(Op.getOperand(0)); - SDValue Op1 = ScalarizeVectorOp(Op.getOperand(1)); - Result = DAG.getNode(ISD::SETCC, dl, - TLI.getSetCCResultType(Op0.getValueType()), - Op0, Op1, Op.getOperand(2)); - Result = DAG.getNode(ISD::SELECT, dl, NewVT, Result, - DAG.getConstant(-1ULL, NewVT), - DAG.getConstant(0ULL, NewVT)); - break; - } - } - - if (TLI.isTypeLegal(NewVT)) - Result = LegalizeOp(Result); - bool isNew = ScalarizedNodes.insert(std::make_pair(Op, Result)).second; - assert(isNew && "Value already scalarized?"); - isNew = isNew; - return Result; -} - - -SDValue SelectionDAGLegalize::WidenVectorOp(SDValue Op, MVT WidenVT) { - std::map<SDValue, SDValue>::iterator I = WidenNodes.find(Op); - if (I != WidenNodes.end()) return I->second; - - MVT VT = Op.getValueType(); - assert(VT.isVector() && "Cannot widen non-vector type!"); - - SDValue Result; - SDNode *Node = Op.getNode(); - DebugLoc dl = Node->getDebugLoc(); - MVT EVT = VT.getVectorElementType(); - - unsigned NumElts = VT.getVectorNumElements(); - unsigned NewNumElts = WidenVT.getVectorNumElements(); - assert(NewNumElts > NumElts && "Cannot widen to smaller type!"); - assert(NewNumElts < 17); - - // When widen is called, it is assumed that it is more efficient to use a - // wide type. The default action is to widen to operation to a wider legal - // vector type and then do the operation if it is legal by calling LegalizeOp - // again. If there is no vector equivalent, we will unroll the operation, do - // it, and rebuild the vector. If most of the operations are vectorizible to - // the legal type, the resulting code will be more efficient. If this is not - // the case, the resulting code will preform badly as we end up generating - // code to pack/unpack the results. It is the function that calls widen - // that is responsible for seeing this doesn't happen. - switch (Node->getOpcode()) { - default: -#ifndef NDEBUG - Node->dump(&DAG); -#endif - assert(0 && "Unexpected operation in WidenVectorOp!"); - break; - case ISD::CopyFromReg: - assert(0 && "CopyFromReg doesn't need widening!"); - case ISD::Constant: - case ISD::ConstantFP: - // To build a vector of these elements, clients should call BuildVector - // and with each element instead of creating a node with a vector type - assert(0 && "Unexpected operation in WidenVectorOp!"); - case ISD::VAARG: - // Variable Arguments with vector types doesn't make any sense to me - assert(0 && "Unexpected operation in WidenVectorOp!"); - break; - case ISD::UNDEF: - Result = DAG.getUNDEF(WidenVT); - break; - case ISD::BUILD_VECTOR: { - // Build a vector with undefined for the new nodes - SDValueVector NewOps(Node->op_begin(), Node->op_end()); - for (unsigned i = NumElts; i < NewNumElts; ++i) { - NewOps.push_back(DAG.getUNDEF(EVT)); - } - Result = DAG.getNode(ISD::BUILD_VECTOR, dl, WidenVT, - &NewOps[0], NewOps.size()); - break; - } - case ISD::INSERT_VECTOR_ELT: { - SDValue Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - Result = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, WidenVT, Tmp1, - Node->getOperand(1), Node->getOperand(2)); - break; - } - case ISD::VECTOR_SHUFFLE: { - SDValue Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - SDValue Tmp2 = WidenVectorOp(Node->getOperand(1), WidenVT); - ShuffleVectorSDNode *SVOp = cast<ShuffleVectorSDNode>(Node); - SmallVector<int, 8> NewMask; - for (unsigned i = 0; i < NumElts; ++i) { - int Idx = SVOp->getMaskElt(i); - if (Idx < (int)NumElts) - NewMask.push_back(Idx); - else - NewMask.push_back(Idx + NewNumElts - NumElts); - } - for (unsigned i = NumElts; i < NewNumElts; ++i) - NewMask.push_back(-1); - - Result = DAG.getVectorShuffle(WidenVT, dl, Tmp1, Tmp2, &NewMask[0]); - break; - } - case ISD::LOAD: { - // If the load widen returns true, we can use a single load for the - // vector. Otherwise, it is returning a token factor for multiple - // loads. - SDValue TFOp; - if (LoadWidenVectorOp(Result, TFOp, Op, WidenVT)) - AddLegalizedOperand(Op.getValue(1), LegalizeOp(TFOp.getValue(1))); - else - AddLegalizedOperand(Op.getValue(1), LegalizeOp(TFOp.getValue(0))); - break; - } - - case ISD::BIT_CONVERT: { - SDValue Tmp1 = Node->getOperand(0); - // Converts between two different types so we need to determine - // the correct widen type for the input operand. - MVT InVT = Tmp1.getValueType(); - unsigned WidenSize = WidenVT.getSizeInBits(); - if (InVT.isVector()) { - MVT InEltVT = InVT.getVectorElementType(); - unsigned InEltSize = InEltVT.getSizeInBits(); - assert(WidenSize % InEltSize == 0 && - "can not widen bit convert that are not multiple of element type"); - MVT NewInWidenVT = MVT::getVectorVT(InEltVT, WidenSize / InEltSize); - Tmp1 = WidenVectorOp(Tmp1, NewInWidenVT); - assert(Tmp1.getValueType().getSizeInBits() == WidenVT.getSizeInBits()); - Result = DAG.getNode(ISD::BIT_CONVERT, dl, WidenVT, Tmp1); - } else { - // If the result size is a multiple of the input size, widen the input - // and then convert. - unsigned InSize = InVT.getSizeInBits(); - assert(WidenSize % InSize == 0 && - "can not widen bit convert that are not multiple of element type"); - unsigned NewNumElts = WidenSize / InSize; - SmallVector<SDValue, 16> Ops(NewNumElts); - SDValue UndefVal = DAG.getUNDEF(InVT); - Ops[0] = Tmp1; - for (unsigned i = 1; i < NewNumElts; ++i) - Ops[i] = UndefVal; - - MVT NewInVT = MVT::getVectorVT(InVT, NewNumElts); - Result = DAG.getNode(ISD::BUILD_VECTOR, dl, NewInVT, &Ops[0], NewNumElts); - Result = DAG.getNode(ISD::BIT_CONVERT, dl, WidenVT, Result); - } - break; - } - - case ISD::SINT_TO_FP: - case ISD::UINT_TO_FP: - case ISD::FP_TO_SINT: - case ISD::FP_TO_UINT: - case ISD::FP_ROUND: { - SDValue Tmp1 = Node->getOperand(0); - // Converts between two different types so we need to determine - // the correct widen type for the input operand. - MVT TVT = Tmp1.getValueType(); - assert(TVT.isVector() && "can not widen non vector type"); - MVT TEVT = TVT.getVectorElementType(); - MVT TWidenVT = MVT::getVectorVT(TEVT, NewNumElts); - Tmp1 = WidenVectorOp(Tmp1, TWidenVT); - assert(Tmp1.getValueType().getVectorNumElements() == NewNumElts); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Tmp1); - break; - } - - case ISD::FP_EXTEND: - assert(0 && "Case not implemented. Dynamically dead with 2 FP types!"); - case ISD::TRUNCATE: - case ISD::SIGN_EXTEND: - case ISD::ZERO_EXTEND: - case ISD::ANY_EXTEND: - case ISD::SIGN_EXTEND_INREG: - case ISD::FABS: - case ISD::FNEG: - case ISD::FSQRT: - case ISD::FSIN: - case ISD::FCOS: - case ISD::CTPOP: - case ISD::CTTZ: - case ISD::CTLZ: { - // Unary op widening - SDValue Tmp1; - Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - assert(Tmp1.getValueType() == WidenVT); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Tmp1); - break; - } - case ISD::CONVERT_RNDSAT: { - SDValue RndOp = Node->getOperand(3); - SDValue SatOp = Node->getOperand(4); - SDValue SrcOp = Node->getOperand(0); - - // Converts between two different types so we need to determine - // the correct widen type for the input operand. - MVT SVT = SrcOp.getValueType(); - assert(SVT.isVector() && "can not widen non vector type"); - MVT SEVT = SVT.getVectorElementType(); - MVT SWidenVT = MVT::getVectorVT(SEVT, NewNumElts); - - SrcOp = WidenVectorOp(SrcOp, SWidenVT); - assert(SrcOp.getValueType() == WidenVT); - SDValue DTyOp = DAG.getValueType(WidenVT); - SDValue STyOp = DAG.getValueType(SrcOp.getValueType()); - ISD::CvtCode CvtCode = cast<CvtRndSatSDNode>(Node)->getCvtCode(); - - Result = DAG.getConvertRndSat(WidenVT, dl, SrcOp, DTyOp, STyOp, - RndOp, SatOp, CvtCode); - break; - } - case ISD::FPOW: - case ISD::FPOWI: - case ISD::ADD: - case ISD::SUB: - case ISD::MUL: - case ISD::MULHS: - case ISD::MULHU: - case ISD::AND: - case ISD::OR: - case ISD::XOR: - case ISD::FADD: - case ISD::FSUB: - case ISD::FMUL: - case ISD::SDIV: - case ISD::SREM: - case ISD::FDIV: - case ISD::FREM: - case ISD::FCOPYSIGN: - case ISD::UDIV: - case ISD::UREM: - case ISD::BSWAP: { - // Binary op widening - SDValue Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - SDValue Tmp2 = WidenVectorOp(Node->getOperand(1), WidenVT); - assert(Tmp1.getValueType() == WidenVT && Tmp2.getValueType() == WidenVT); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Tmp1, Tmp2); - break; - } - - case ISD::SHL: - case ISD::SRA: - case ISD::SRL: { - SDValue Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - assert(Tmp1.getValueType() == WidenVT); - SDValue ShOp = Node->getOperand(1); - MVT ShVT = ShOp.getValueType(); - MVT NewShVT = MVT::getVectorVT(ShVT.getVectorElementType(), - WidenVT.getVectorNumElements()); - ShOp = WidenVectorOp(ShOp, NewShVT); - assert(ShOp.getValueType() == NewShVT); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Tmp1, ShOp); - break; - } - - case ISD::EXTRACT_VECTOR_ELT: { - SDValue Tmp1 = WidenVectorOp(Node->getOperand(0), WidenVT); - assert(Tmp1.getValueType() == WidenVT); - Result = DAG.getNode(Node->getOpcode(), dl, EVT, Tmp1, Node->getOperand(1)); - break; - } - case ISD::CONCAT_VECTORS: { - // We concurrently support only widen on a multiple of the incoming vector. - // We could widen on a multiple of the incoming operand if necessary. - unsigned NumConcat = NewNumElts / NumElts; - assert(NewNumElts % NumElts == 0 && "Can widen only a multiple of vector"); - SDValue UndefVal = DAG.getUNDEF(VT); - SmallVector<SDValue, 8> MOps; - MOps.push_back(Op); - for (unsigned i = 1; i != NumConcat; ++i) { - MOps.push_back(UndefVal); - } - Result = LegalizeOp(DAG.getNode(ISD::CONCAT_VECTORS, dl, WidenVT, - &MOps[0], MOps.size())); - break; - } - case ISD::EXTRACT_SUBVECTOR: { - SDValue Tmp1 = Node->getOperand(0); - SDValue Idx = Node->getOperand(1); - ConstantSDNode *CIdx = dyn_cast<ConstantSDNode>(Idx); - if (CIdx && CIdx->getZExtValue() == 0) { - // Since we are access the start of the vector, the incoming - // vector type might be the proper. - MVT Tmp1VT = Tmp1.getValueType(); - if (Tmp1VT == WidenVT) - return Tmp1; - else { - unsigned Tmp1VTNumElts = Tmp1VT.getVectorNumElements(); - if (Tmp1VTNumElts < NewNumElts) - Result = WidenVectorOp(Tmp1, WidenVT); - else - Result = DAG.getNode(ISD::EXTRACT_SUBVECTOR, dl, WidenVT, Tmp1, Idx); - } - } else if (NewNumElts % NumElts == 0) { - // Widen the extracted subvector. - unsigned NumConcat = NewNumElts / NumElts; - SDValue UndefVal = DAG.getUNDEF(VT); - SmallVector<SDValue, 8> MOps; - MOps.push_back(Op); - for (unsigned i = 1; i != NumConcat; ++i) { - MOps.push_back(UndefVal); - } - Result = LegalizeOp(DAG.getNode(ISD::CONCAT_VECTORS, dl, WidenVT, - &MOps[0], MOps.size())); - } else { - assert(0 && "can not widen extract subvector"); - // This could be implemented using insert and build vector but I would - // like to see when this happens. - } - break; - } - - case ISD::SELECT: { - // Determine new condition widen type and widen - SDValue Cond1 = Node->getOperand(0); - MVT CondVT = Cond1.getValueType(); - assert(CondVT.isVector() && "can not widen non vector type"); - MVT CondEVT = CondVT.getVectorElementType(); - MVT CondWidenVT = MVT::getVectorVT(CondEVT, NewNumElts); - Cond1 = WidenVectorOp(Cond1, CondWidenVT); - assert(Cond1.getValueType() == CondWidenVT && "Condition not widen"); - - SDValue Tmp1 = WidenVectorOp(Node->getOperand(1), WidenVT); - SDValue Tmp2 = WidenVectorOp(Node->getOperand(2), WidenVT); - assert(Tmp1.getValueType() == WidenVT && Tmp2.getValueType() == WidenVT); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Cond1, Tmp1, Tmp2); - break; - } - - case ISD::SELECT_CC: { - // Determine new condition widen type and widen - SDValue Cond1 = Node->getOperand(0); - SDValue Cond2 = Node->getOperand(1); - MVT CondVT = Cond1.getValueType(); - assert(CondVT.isVector() && "can not widen non vector type"); - assert(CondVT == Cond2.getValueType() && "mismatch lhs/rhs"); - MVT CondEVT = CondVT.getVectorElementType(); - MVT CondWidenVT = MVT::getVectorVT(CondEVT, NewNumElts); - Cond1 = WidenVectorOp(Cond1, CondWidenVT); - Cond2 = WidenVectorOp(Cond2, CondWidenVT); - assert(Cond1.getValueType() == CondWidenVT && - Cond2.getValueType() == CondWidenVT && "condition not widen"); - - SDValue Tmp1 = WidenVectorOp(Node->getOperand(2), WidenVT); - SDValue Tmp2 = WidenVectorOp(Node->getOperand(3), WidenVT); - assert(Tmp1.getValueType() == WidenVT && Tmp2.getValueType() == WidenVT && - "operands not widen"); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Cond1, Cond2, Tmp1, - Tmp2, Node->getOperand(4)); - break; - } - case ISD::VSETCC: { - // Determine widen for the operand - SDValue Tmp1 = Node->getOperand(0); - MVT TmpVT = Tmp1.getValueType(); - assert(TmpVT.isVector() && "can not widen non vector type"); - MVT TmpEVT = TmpVT.getVectorElementType(); - MVT TmpWidenVT = MVT::getVectorVT(TmpEVT, NewNumElts); - Tmp1 = WidenVectorOp(Tmp1, TmpWidenVT); - SDValue Tmp2 = WidenVectorOp(Node->getOperand(1), TmpWidenVT); - Result = DAG.getNode(Node->getOpcode(), dl, WidenVT, Tmp1, Tmp2, - Node->getOperand(2)); - break; - } - case ISD::ATOMIC_CMP_SWAP: - case ISD::ATOMIC_LOAD_ADD: - case ISD::ATOMIC_LOAD_SUB: - case ISD::ATOMIC_LOAD_AND: - case ISD::ATOMIC_LOAD_OR: - case ISD::ATOMIC_LOAD_XOR: - case ISD::ATOMIC_LOAD_NAND: - case ISD::ATOMIC_LOAD_MIN: - case ISD::ATOMIC_LOAD_MAX: - case ISD::ATOMIC_LOAD_UMIN: - case ISD::ATOMIC_LOAD_UMAX: - case ISD::ATOMIC_SWAP: { - // For now, we assume that using vectors for these operations don't make - // much sense so we just split it. We return an empty result - SDValue X, Y; - SplitVectorOp(Op, X, Y); - return Result; - break; - } - - } // end switch (Node->getOpcode()) - - assert(Result.getNode() && "Didn't set a result!"); - if (Result != Op) - Result = LegalizeOp(Result); - - AddWidenedOperand(Op, Result); - return Result; -} - -// Utility function to find a legal vector type and its associated element -// type from a preferred width and whose vector type must be the same size -// as the VVT. -// TLI: Target lowering used to determine legal types -// Width: Preferred width of element type -// VVT: Vector value type whose size we must match. -// Returns VecEVT and EVT - the vector type and its associated element type -static void FindWidenVecType(const TargetLowering &TLI, unsigned Width, MVT VVT, - MVT& EVT, MVT& VecEVT) { - // We start with the preferred width, make it a power of 2 and see if - // we can find a vector type of that width. If not, we reduce it by - // another power of 2. If we have widen the type, a vector of bytes should - // always be legal. - assert(TLI.isTypeLegal(VVT)); - unsigned EWidth = Width + 1; - do { - assert(EWidth > 0); - EWidth = (1 << Log2_32(EWidth-1)); - EVT = MVT::getIntegerVT(EWidth); - unsigned NumEVT = VVT.getSizeInBits()/EWidth; - VecEVT = MVT::getVectorVT(EVT, NumEVT); - } while (!TLI.isTypeLegal(VecEVT) || - VVT.getSizeInBits() != VecEVT.getSizeInBits()); -} - -SDValue SelectionDAGLegalize::genWidenVectorLoads(SDValueVector& LdChain, - SDValue Chain, - SDValue BasePtr, - const Value *SV, - int SVOffset, - unsigned Alignment, - bool isVolatile, - unsigned LdWidth, - MVT ResType, - DebugLoc dl) { - // We assume that we have good rules to handle loading power of two loads so - // we break down the operations to power of 2 loads. The strategy is to - // load the largest power of 2 that we can easily transform to a legal vector - // and then insert into that vector, and the cast the result into the legal - // vector that we want. This avoids unnecessary stack converts. - // TODO: If the Ldwidth is legal, alignment is the same as the LdWidth, and - // the load is nonvolatile, we an use a wider load for the value. - // Find a vector length we can load a large chunk - MVT EVT, VecEVT; - unsigned EVTWidth; - FindWidenVecType(TLI, LdWidth, ResType, EVT, VecEVT); - EVTWidth = EVT.getSizeInBits(); - - SDValue LdOp = DAG.getLoad(EVT, dl, Chain, BasePtr, SV, SVOffset, - isVolatile, Alignment); - SDValue VecOp = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VecEVT, LdOp); - LdChain.push_back(LdOp.getValue(1)); - - // Check if we can load the element with one instruction - if (LdWidth == EVTWidth) { - return DAG.getNode(ISD::BIT_CONVERT, dl, ResType, VecOp); - } - - // The vector element order is endianness dependent. - unsigned Idx = 1; - LdWidth -= EVTWidth; - unsigned Offset = 0; - - while (LdWidth > 0) { - unsigned Increment = EVTWidth / 8; - Offset += Increment; - BasePtr = DAG.getNode(ISD::ADD, dl, BasePtr.getValueType(), BasePtr, - DAG.getIntPtrConstant(Increment)); - - if (LdWidth < EVTWidth) { - // Our current type we are using is too large, use a smaller size by - // using a smaller power of 2 - unsigned oEVTWidth = EVTWidth; - FindWidenVecType(TLI, LdWidth, ResType, EVT, VecEVT); - EVTWidth = EVT.getSizeInBits(); - // Readjust position and vector position based on new load type - Idx = Idx * (oEVTWidth/EVTWidth); - VecOp = DAG.getNode(ISD::BIT_CONVERT, dl, VecEVT, VecOp); - } - - SDValue LdOp = DAG.getLoad(EVT, dl, Chain, BasePtr, SV, - SVOffset+Offset, isVolatile, - MinAlign(Alignment, Offset)); - LdChain.push_back(LdOp.getValue(1)); - VecOp = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, VecEVT, VecOp, LdOp, - DAG.getIntPtrConstant(Idx++)); - - LdWidth -= EVTWidth; - } - - return DAG.getNode(ISD::BIT_CONVERT, dl, ResType, VecOp); -} - -bool SelectionDAGLegalize::LoadWidenVectorOp(SDValue& Result, - SDValue& TFOp, - SDValue Op, - MVT NVT) { - // TODO: Add support for ConcatVec and the ability to load many vector - // types (e.g., v4i8). This will not work when a vector register - // to memory mapping is strange (e.g., vector elements are not - // stored in some sequential order). - - // It must be true that the widen vector type is bigger than where - // we need to load from. - LoadSDNode *LD = cast<LoadSDNode>(Op.getNode()); - MVT LdVT = LD->getMemoryVT(); - DebugLoc dl = LD->getDebugLoc(); - assert(LdVT.isVector() && NVT.isVector()); - assert(LdVT.getVectorElementType() == NVT.getVectorElementType()); - - // Load information - SDValue Chain = LD->getChain(); - SDValue BasePtr = LD->getBasePtr(); - int SVOffset = LD->getSrcValueOffset(); - unsigned Alignment = LD->getAlignment(); - bool isVolatile = LD->isVolatile(); - const Value *SV = LD->getSrcValue(); - unsigned int LdWidth = LdVT.getSizeInBits(); - - // Load value as a large register - SDValueVector LdChain; - Result = genWidenVectorLoads(LdChain, Chain, BasePtr, SV, SVOffset, - Alignment, isVolatile, LdWidth, NVT, dl); - - if (LdChain.size() == 1) { - TFOp = LdChain[0]; - return true; - } - else { - TFOp=DAG.getNode(ISD::TokenFactor, dl, MVT::Other, - &LdChain[0], LdChain.size()); - return false; - } -} - - -void SelectionDAGLegalize::genWidenVectorStores(SDValueVector& StChain, - SDValue Chain, - SDValue BasePtr, - const Value *SV, - int SVOffset, - unsigned Alignment, - bool isVolatile, - SDValue ValOp, - unsigned StWidth, - DebugLoc dl) { - // Breaks the stores into a series of power of 2 width stores. For any - // width, we convert the vector to the vector of element size that we - // want to store. This avoids requiring a stack convert. - - // Find a width of the element type we can store with - MVT VVT = ValOp.getValueType(); - MVT EVT, VecEVT; - unsigned EVTWidth; - FindWidenVecType(TLI, StWidth, VVT, EVT, VecEVT); - EVTWidth = EVT.getSizeInBits(); - - SDValue VecOp = DAG.getNode(ISD::BIT_CONVERT, dl, VecEVT, ValOp); - SDValue EOp = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, EVT, VecOp, - DAG.getIntPtrConstant(0)); - SDValue StOp = DAG.getStore(Chain, dl, EOp, BasePtr, SV, SVOffset, - isVolatile, Alignment); - StChain.push_back(StOp); - - // Check if we are done - if (StWidth == EVTWidth) { - return; - } - - unsigned Idx = 1; - StWidth -= EVTWidth; - unsigned Offset = 0; - - while (StWidth > 0) { - unsigned Increment = EVTWidth / 8; - Offset += Increment; - BasePtr = DAG.getNode(ISD::ADD, dl, BasePtr.getValueType(), BasePtr, - DAG.getIntPtrConstant(Increment)); - - if (StWidth < EVTWidth) { - // Our current type we are using is too large, use a smaller size by - // using a smaller power of 2 - unsigned oEVTWidth = EVTWidth; - FindWidenVecType(TLI, StWidth, VVT, EVT, VecEVT); - EVTWidth = EVT.getSizeInBits(); - // Readjust position and vector position based on new load type - Idx = Idx * (oEVTWidth/EVTWidth); - VecOp = DAG.getNode(ISD::BIT_CONVERT, dl, VecEVT, VecOp); - } - - EOp = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, EVT, VecOp, - DAG.getIntPtrConstant(Idx++)); - StChain.push_back(DAG.getStore(Chain, dl, EOp, BasePtr, SV, - SVOffset + Offset, isVolatile, - MinAlign(Alignment, Offset))); - StWidth -= EVTWidth; - } -} - - -SDValue SelectionDAGLegalize::StoreWidenVectorOp(StoreSDNode *ST, - SDValue Chain, - SDValue BasePtr) { - // TODO: It might be cleaner if we can use SplitVector and have more legal - // vector types that can be stored into memory (e.g., v4xi8 can - // be stored as a word). This will not work when a vector register - // to memory mapping is strange (e.g., vector elements are not - // stored in some sequential order). - - MVT StVT = ST->getMemoryVT(); - SDValue ValOp = ST->getValue(); - DebugLoc dl = ST->getDebugLoc(); - - // Check if we have widen this node with another value - std::map<SDValue, SDValue>::iterator I = WidenNodes.find(ValOp); - if (I != WidenNodes.end()) - ValOp = I->second; - - MVT VVT = ValOp.getValueType(); - - // It must be true that we the widen vector type is bigger than where - // we need to store. - assert(StVT.isVector() && VVT.isVector()); - assert(StVT.bitsLT(VVT)); - assert(StVT.getVectorElementType() == VVT.getVectorElementType()); - - // Store value - SDValueVector StChain; - genWidenVectorStores(StChain, Chain, BasePtr, ST->getSrcValue(), - ST->getSrcValueOffset(), ST->getAlignment(), - ST->isVolatile(), ValOp, StVT.getSizeInBits(), dl); - if (StChain.size() == 1) - return StChain[0]; - else - return DAG.getNode(ISD::TokenFactor, dl, MVT::Other, - &StChain[0], StChain.size()); -} - - // SelectionDAG::Legalize - This is the entry point for the file. // void SelectionDAG::Legalize(bool TypesNeedLegalizing, Index: lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp ==================================================================--- lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp (revision 72159) +++ lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp (working copy) @@ -611,6 +611,36 @@ DOUT << "Optimized type-legalized selection DAG:\n"; DEBUG(CurDAG->dump()); } + + if (TimePassesIsEnabled) { + NamedRegionTimer T("Vector Legalization", GroupName); + Changed = CurDAG->LegalizeVectors(); + } else { + Changed = CurDAG->LegalizeVectors(); + } + + if (Changed) { + if (TimePassesIsEnabled) { + NamedRegionTimer T("Type Legalization 2", GroupName); + Changed = CurDAG->LegalizeTypes(); + } else { + Changed = CurDAG->LegalizeTypes(); + } + + if (ViewDAGCombineLT) + CurDAG->viewGraph("dag-combine-lv input for " + BlockName); + + // Run the DAG combiner in post-type-legalize mode. + if (TimePassesIsEnabled) { + NamedRegionTimer T("DAG Combining after legalize vectors", GroupName); + CurDAG->Combine(NoIllegalOperations, *AA, OptLevel); + } else { + CurDAG->Combine(NoIllegalOperations, *AA, OptLevel); + } + + DOUT << "Optimized vector-legalized selection DAG:\n"; + DEBUG(CurDAG->dump()); + } } if (ViewLegalizeDAGs) CurDAG->viewGraph("legalize input for " + BlockName);
Apparently Analagous Threads
- [LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
- [LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
- [LLVMdev] [PATCH] Add new phase to legalization to handle vector operations
- [LLVMdev] Possible bug in ExpandShiftWithUnknownAmountBit
- [LLVMdev] [PATCH] Add new phase to legalization to handle vector operations