[VPlan] Fold VPDef into VPRecipeBase (NFC). #174282
Open
+275
−312
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
@llvm/pr-subscribers-llvm-transforms Author: Florian Hahn (fhahn) ChangesA separate VDef is not needed any longer, fold i into VPRecipeBase to Depends on #172758. Patch is 117.76 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/174282.diff 16 Files Affected:
diff --git a/llvm/docs/VectorizationPlan.rst b/llvm/docs/VectorizationPlan.rst
index 73e9e6098175a..b8db8cab843cf 100644
--- a/llvm/docs/VectorizationPlan.rst
+++ b/llvm/docs/VectorizationPlan.rst
@@ -186,7 +186,8 @@ The low-level design of VPlan comprises of the following classes.
input IR instructions are referred to as "Ingredients" of the Recipe. A Recipe
may specify how its ingredients are to be transformed to produce the output IR
instructions; e.g., cloned once, replicated multiple times or widened
- according to selected VF.
+ according to selected VF. VPRecipeBase also defines zero, one or multiple
+ VPValues, modeling the fact that recipes can produce multiple results.
:VPValue:
The base of VPlan's def-use relations class hierarchy. When instantiated, it
@@ -197,11 +198,6 @@ The low-level design of VPlan comprises of the following classes.
A VPUser represents an entity that uses a number of VPValues as operands.
VPUser is similar in some aspects to LLVM's User class.
-:VPDef:
- A VPDef represents an entity that defines zero, one or multiple VPValues.
- It is used to model the fact that recipes in VPlan can define multiple
- VPValues.
-
:VPInstruction:
A VPInstruction is a recipe characterized by a single opcode and optional
flags, free of ingredients or other meta-data. VPInstructions also extend
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 430d9469c7698..ddef767f6937c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -296,7 +296,7 @@ class VPBuilder {
/// induction with \p Start and \p Step values, using \p Start + \p Current *
/// \p Step.
VPDerivedIVRecipe *createDerivedIV(InductionDescriptor::InductionKind Kind,
- FPMathOperator *FPBinOp, VPValue *Start,
+ FPMathOperator *FPBinOp, VPIRValue *Start,
VPValue *Current, VPValue *Step,
const Twine &Name = "") {
return tryInsertInstruction(
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 04e08e0349074..4ce686bb6cf3d 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -4110,41 +4110,41 @@ static bool willGenerateVectors(VPlan &Plan, ElementCount VF,
// result. Note that this includes VPInstruction where some opcodes may
// produce a vector, to preserve existing behavior as VPInstructions model
// aspects not directly mapped to existing IR instructions.
- switch (R.getVPDefID()) {
- case VPDef::VPDerivedIVSC:
- case VPDef::VPScalarIVStepsSC:
- case VPDef::VPReplicateSC:
- case VPDef::VPInstructionSC:
- case VPDef::VPCanonicalIVPHISC:
- case VPDef::VPVectorPointerSC:
- case VPDef::VPVectorEndPointerSC:
- case VPDef::VPExpandSCEVSC:
- case VPDef::VPEVLBasedIVPHISC:
- case VPDef::VPPredInstPHISC:
- case VPDef::VPBranchOnMaskSC:
+ switch (R.getVPRecipeID()) {
+ case VPRecipeBase::VPDerivedIVSC:
+ case VPRecipeBase::VPScalarIVStepsSC:
+ case VPRecipeBase::VPReplicateSC:
+ case VPRecipeBase::VPInstructionSC:
+ case VPRecipeBase::VPCanonicalIVPHISC:
+ case VPRecipeBase::VPVectorPointerSC:
+ case VPRecipeBase::VPVectorEndPointerSC:
+ case VPRecipeBase::VPExpandSCEVSC:
+ case VPRecipeBase::VPEVLBasedIVPHISC:
+ case VPRecipeBase::VPPredInstPHISC:
+ case VPRecipeBase::VPBranchOnMaskSC:
continue;
- case VPDef::VPReductionSC:
- case VPDef::VPActiveLaneMaskPHISC:
- case VPDef::VPWidenCallSC:
- case VPDef::VPWidenCanonicalIVSC:
- case VPDef::VPWidenCastSC:
- case VPDef::VPWidenGEPSC:
- case VPDef::VPWidenIntrinsicSC:
- case VPDef::VPWidenSC:
- case VPDef::VPWidenSelectSC:
- case VPDef::VPBlendSC:
- case VPDef::VPFirstOrderRecurrencePHISC:
- case VPDef::VPHistogramSC:
- case VPDef::VPWidenPHISC:
- case VPDef::VPWidenIntOrFpInductionSC:
- case VPDef::VPWidenPointerInductionSC:
- case VPDef::VPReductionPHISC:
- case VPDef::VPInterleaveEVLSC:
- case VPDef::VPInterleaveSC:
- case VPDef::VPWidenLoadEVLSC:
- case VPDef::VPWidenLoadSC:
- case VPDef::VPWidenStoreEVLSC:
- case VPDef::VPWidenStoreSC:
+ case VPRecipeBase::VPReductionSC:
+ case VPRecipeBase::VPActiveLaneMaskPHISC:
+ case VPRecipeBase::VPWidenCallSC:
+ case VPRecipeBase::VPWidenCanonicalIVSC:
+ case VPRecipeBase::VPWidenCastSC:
+ case VPRecipeBase::VPWidenGEPSC:
+ case VPRecipeBase::VPWidenIntrinsicSC:
+ case VPRecipeBase::VPWidenSC:
+ case VPRecipeBase::VPWidenSelectSC:
+ case VPRecipeBase::VPBlendSC:
+ case VPRecipeBase::VPFirstOrderRecurrencePHISC:
+ case VPRecipeBase::VPHistogramSC:
+ case VPRecipeBase::VPWidenPHISC:
+ case VPRecipeBase::VPWidenIntOrFpInductionSC:
+ case VPRecipeBase::VPWidenPointerInductionSC:
+ case VPRecipeBase::VPReductionPHISC:
+ case VPRecipeBase::VPInterleaveEVLSC:
+ case VPRecipeBase::VPInterleaveSC:
+ case VPRecipeBase::VPWidenLoadEVLSC:
+ case VPRecipeBase::VPWidenLoadSC:
+ case VPRecipeBase::VPWidenStoreEVLSC:
+ case VPRecipeBase::VPWidenStoreSC:
break;
default:
llvm_unreachable("unhandled recipe");
@@ -7424,11 +7424,12 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// making any changes to the CFG.
DenseMap<const SCEV *, Value *> ExpandedSCEVs =
VPlanTransforms::expandSCEVs(BestVPlan, *PSE.getSE());
- if (!ILV.getTripCount())
+ if (!ILV.getTripCount()) {
ILV.setTripCount(BestVPlan.getTripCount()->getLiveInIRValue());
- else
+ } else {
assert(VectorizingEpilogue && "should only re-use the existing trip "
"count during epilogue vectorization");
+ }
// Perform the actual loop transformation.
VPTransformState State(&TTI, BestVF, LI, DT, ILV.AC, ILV.Builder, &BestVPlan,
@@ -7747,7 +7748,7 @@ VPRecipeBuilder::tryToOptimizeInductionTruncate(VPInstruction *VPI,
auto *WidenIV = cast<VPWidenIntOrFpInductionRecipe>(
VPI->getOperand(0)->getDefiningRecipe());
PHINode *Phi = WidenIV->getPHINode();
- VPValue *Start = WidenIV->getStartValue();
+ VPIRValue *Start = WidenIV->getStartValue();
const InductionDescriptor &IndDesc = WidenIV->getInductionDescriptor();
// It is always safe to copy over the NoWrap and FastMath flags. In
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 6dd250355ac34..f25e76b4015e8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -91,18 +91,6 @@ Value *VPLane::getAsRuntimeExpr(IRBuilderBase &Builder,
llvm_unreachable("Unknown lane kind");
}
-VPValue::VPValue(const unsigned char SC, Value *UV, VPDef *Def)
- : SubclassID(SC), UnderlyingVal(UV), Def(Def) {
- if (Def)
- Def->addDefinedValue(this);
-}
-
-VPValue::~VPValue() {
- assert(Users.empty() && "trying to delete a VPValue with remaining users");
- if (VPDef *Def = getDefiningRecipe())
- Def->removeDefinedValue(this);
-}
-
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
void VPValue::print(raw_ostream &OS, VPSlotTracker &SlotTracker) const {
if (const VPRecipeBase *R = getDefiningRecipe())
@@ -119,21 +107,43 @@ void VPValue::dump() const {
dbgs() << "\n";
}
-void VPDef::dump() const {
- const VPRecipeBase *Instr = dyn_cast_or_null<VPRecipeBase>(this);
- VPSlotTracker SlotTracker(
- (Instr && Instr->getParent()) ? Instr->getParent()->getPlan() : nullptr);
+void VPRecipeBase::dump() const {
+ VPSlotTracker SlotTracker(getParent() ? getParent()->getPlan() : nullptr);
print(dbgs(), "", SlotTracker);
dbgs() << "\n";
}
#endif
VPRecipeBase *VPValue::getDefiningRecipe() {
- return cast_or_null<VPRecipeBase>(Def);
+ auto *DefValue = dyn_cast<VPRecipeValue>(this);
+ if (!DefValue)
+ return nullptr;
+ return DefValue->Def;
}
const VPRecipeBase *VPValue::getDefiningRecipe() const {
- return cast_or_null<VPRecipeBase>(Def);
+ auto *DefValue = dyn_cast<VPRecipeValue>(this);
+ if (!DefValue)
+ return nullptr;
+ return DefValue->Def;
+}
+
+Value *VPValue::getLiveInIRValue() const {
+ return cast<VPIRValue>(this)->getValue();
+}
+
+Type *VPIRValue::getType() const { return getUnderlyingValue()->getType(); }
+
+VPRecipeValue::VPRecipeValue(VPRecipeBase *Def, Value *UV)
+ : VPValue(VPVRecipeValueSC, UV), Def(Def) {
+ assert(Def && "VPRecipeValue requires a defining recipe");
+ Def->addDefinedValue(this);
+}
+
+VPRecipeValue::~VPRecipeValue() {
+ assert(Users.empty() &&
+ "trying to delete a VPRecipeValue with remaining users");
+ Def->removeDefinedValue(this);
}
// Get the top-most entry block of \p Start. This is the entry block of the
@@ -229,8 +239,8 @@ VPTransformState::VPTransformState(const TargetTransformInfo *TTI,
CurrentParentLoop(CurrentParentLoop), TypeAnalysis(*Plan), VPDT(*Plan) {}
Value *VPTransformState::get(const VPValue *Def, const VPLane &Lane) {
- if (Def->isLiveIn())
- return Def->getLiveInIRValue();
+ if (isa<VPIRValue, VPSymbolicValue>(Def))
+ return Def->getUnderlyingValue();
if (hasScalarValue(Def, Lane))
return Data.VPV2Scalars[Def][Lane.mapToCacheIndex(VF)];
@@ -262,8 +272,8 @@ Value *VPTransformState::get(const VPValue *Def, const VPLane &Lane) {
Value *VPTransformState::get(const VPValue *Def, bool NeedsScalar) {
if (NeedsScalar) {
- assert((VF.isScalar() || Def->isLiveIn() || hasVectorValue(Def) ||
- !vputils::onlyFirstLaneUsed(Def) ||
+ assert((VF.isScalar() || isa<VPIRValue, VPSymbolicValue>(Def) ||
+ hasVectorValue(Def) || !vputils::onlyFirstLaneUsed(Def) ||
(hasScalarValue(Def, VPLane(0)) &&
Data.VPV2Scalars[Def].size() == 1)) &&
"Trying to access a single scalar per part but has multiple scalars "
@@ -284,7 +294,6 @@ Value *VPTransformState::get(const VPValue *Def, bool NeedsScalar) {
};
if (!hasScalarValue(Def, {0})) {
- assert(Def->isLiveIn() && "expected a live-in");
Value *IRV = Def->getLiveInIRValue();
Value *B = GetBroadcastInstrs(IRV);
set(Def, B);
@@ -866,7 +875,7 @@ VPlan::VPlan(Loop *L) {
}
VPlan::~VPlan() {
- VPValue DummyValue;
+ VPSymbolicValue DummyValue;
for (auto *VPB : CreatedBlocks) {
if (auto *VPBB = dyn_cast<VPBasicBlock>(VPB)) {
@@ -1053,7 +1062,7 @@ void VPlan::printLiveIns(raw_ostream &O) const {
O << "\n";
if (TripCount) {
- if (TripCount->isLiveIn())
+ if (isa<VPIRValue>(TripCount))
O << "Live-in ";
TripCount->printAsOperand(O, SlotTracker);
O << " = original trip-count";
@@ -1169,20 +1178,17 @@ VPlan *VPlan::duplicate() {
// Create VPlan, clone live-ins and remap operands in the cloned blocks.
auto *NewPlan = new VPlan(cast<VPBasicBlock>(NewEntry), NewScalarHeader);
DenseMap<VPValue *, VPValue *> Old2NewVPValues;
- for (VPValue *OldLiveIn : getLiveIns()) {
- Old2NewVPValues[OldLiveIn] =
- NewPlan->getOrAddLiveIn(OldLiveIn->getLiveInIRValue());
- }
+ for (VPIRValue *OldLiveIn : getLiveIns())
+ Old2NewVPValues[OldLiveIn] = NewPlan->getOrAddLiveIn(OldLiveIn->getValue());
Old2NewVPValues[&VectorTripCount] = &NewPlan->VectorTripCount;
Old2NewVPValues[&VF] = &NewPlan->VF;
Old2NewVPValues[&VFxUF] = &NewPlan->VFxUF;
if (BackedgeTakenCount) {
- NewPlan->BackedgeTakenCount = new VPValue();
+ NewPlan->BackedgeTakenCount = new VPSymbolicValue();
Old2NewVPValues[BackedgeTakenCount] = NewPlan->BackedgeTakenCount;
}
- if (TripCount && TripCount->isLiveIn())
- Old2NewVPValues[TripCount] =
- NewPlan->getOrAddLiveIn(TripCount->getLiveInIRValue());
+ if (auto *LI = dyn_cast_or_null<VPIRValue>(TripCount))
+ Old2NewVPValues[LI] = NewPlan->getOrAddLiveIn(LI->getValue());
// else NewTripCount will be created and inserted into Old2NewVPValues when
// TripCount is cloned. In any case NewPlan->TripCount is updated below.
@@ -1450,7 +1456,7 @@ void VPSlotTracker::assignName(const VPValue *V) {
const auto &[A, _] = VPValue2Name.try_emplace(V, BaseName);
// Integer or FP constants with different types will result in he same string
// due to stripping types.
- if (V->isLiveIn() && isa<ConstantInt, ConstantFP>(UV))
+ if (isa<VPIRValue>(V) && isa<ConstantInt, ConstantFP>(UV))
return;
// If it is already used by C > 0 other VPValues, increase the version counter
@@ -1727,10 +1733,10 @@ bool llvm::canConstantBeExtended(const APInt *C, Type *NarrowType,
TargetTransformInfo::OperandValueInfo
VPCostContext::getOperandInfo(VPValue *V) const {
- if (!V->isLiveIn())
- return {};
+ if (auto *LI = dyn_cast<VPIRValue>(V))
+ return TTI::getOperandInfo(LI->getValue());
- return TTI::getOperandInfo(V->getLiveInIRValue());
+ return {};
}
InstructionCost VPCostContext::getScalarizationOverhead(
@@ -1758,7 +1764,7 @@ InstructionCost VPCostContext::getScalarizationOverhead(
SmallPtrSet<const VPValue *, 4> UniqueOperands;
SmallVector<Type *> Tys;
for (auto *Op : Operands) {
- if (Op->isLiveIn() ||
+ if (isa<VPIRValue>(Op) ||
(!AlwaysIncludeReplicatingR &&
isa<VPReplicateRecipe, VPPredInstPHIRecipe>(Op)) ||
(isa<VPReplicateRecipe>(Op) &&
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index e2dfc4678c6d0..a59bf2c6b444d 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -377,16 +377,16 @@ class LLVM_ABI_FOR_TEST VPBlockBase {
};
/// VPRecipeBase is a base class modeling a sequence of one or more output IR
-/// instructions. VPRecipeBase owns the VPValues it defines through VPDef
-/// and is responsible for deleting its defined values. Single-value
-/// recipes must inherit from VPSingleDef instead of inheriting from both
-/// VPRecipeBase and VPValue separately.
+/// instructions. VPRecipeBase owns the VPValues it defines and is responsible
+/// for deleting its defined values. Single-value recipes must inherit from
+/// VPSingleDefRecipe instead of inheriting from both VPRecipeBase and VPValue
+/// separately.
class LLVM_ABI_FOR_TEST VPRecipeBase
: public ilist_node_with_parent<VPRecipeBase, VPBasicBlock>,
- public VPDef,
public VPUser {
friend VPBasicBlock;
friend class VPBlockUtils;
+ friend class VPRecipeValue;
/// Each VPRecipe belongs to a single VPBasicBlock.
VPBasicBlock *Parent = nullptr;
@@ -394,12 +394,133 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
/// The debug location for the recipe.
DebugLoc DL;
+ /// Subclass identifier (for isa/dyn_cast).
+ const unsigned char SubclassID;
+
+ /// The VPValues defined by this VPRecipeBase.
+ TinyPtrVector<VPRecipeValue *> DefinedValues;
+
+ /// Add \p V as a defined value by this VPRecipeBase.
+ void addDefinedValue(VPRecipeValue *V) {
+ assert(V->Def == this &&
+ "can only add VPValue already linked with this VPRecipeBase");
+ DefinedValues.push_back(V);
+ }
+
+ /// Remove \p V from the values defined by this VPRecipeBase. \p V must be a
+ /// defined value of this VPRecipeBase.
+ void removeDefinedValue(VPRecipeValue *V) {
+ assert(V->Def == this &&
+ "can only remove VPValue linked with this VPRecipeBase");
+ assert(is_contained(DefinedValues, V) &&
+ "VPValue to remove must be in DefinedValues");
+ llvm::erase(DefinedValues, V);
+ V->Def = nullptr;
+ }
+
public:
+ /// An enumeration for keeping track of the concrete subclass of VPRecipeBase
+ /// that is actually instantiated. Values of this enumeration are kept in the
+ /// SubclassID field of the VPRecipeBase objects. They are used for concrete
+ /// type identification.
+ using VPRecipeTy = enum {
+ VPBranchOnMaskSC,
+ VPDerivedIVSC,
+ VPExpandSCEVSC,
+ VPExpressionSC,
+ VPIRInstructionSC,
+ VPInstructionSC,
+ VPInterleaveEVLSC,
+ VPInterleaveSC,
+ VPReductionEVLSC,
+ VPReductionSC,
+ VPReplicateSC,
+ VPScalarIVStepsSC,
+ VPVectorPointerSC,
+ VPVectorEndPointerSC,
+ VPWidenCallSC,
+ VPWidenCanonicalIVSC,
+ VPWidenCastSC,
+ VPWidenGEPSC,
+ VPWidenIntrinsicSC,
+ VPWidenLoadEVLSC,
+ VPWidenLoadSC,
+ VPWidenStoreEVLSC,
+ VPWidenStoreSC,
+ VPWidenSC,
+ VPWidenSelectSC,
+ VPBlendSC,
+ VPHistogramSC,
+ // START: Phi-like recipes. Need to be kept together.
+ VPWidenPHISC,
+ VPPredInstPHISC,
+ // START: SubclassID for recipes that inherit VPHeaderPHIRecipe.
+ // VPHeaderPHIRecipe need to be kept together.
+ VPCanonicalIVPHISC,
+ VPActiveLaneMaskPHISC,
+ VPEVLBasedIVPHISC,
+ VPFirstOrderRecurrencePHISC,
+ VPWidenIntOrFpInductionSC,
+ VPWidenPointerInductionSC,
+ VPReductionPHISC,
+ // END: SubclassID for recipes that inherit VPHeaderPHIRecipe
+ // END: Phi-like recipes
+ VPFirstPHISC = VPWidenPHISC,
+ VPFirstHeaderPHISC = VPCanonicalIVPHISC,
+ VPLastHeaderPHISC = VPReductionPHISC,
+ VPLastPHISC = VPReductionPHISC,
+ };
+
VPRecipeBase(const unsigned char SC, ArrayRef<VPValue *> Operands,
DebugLoc DL = DebugLoc::getUnknown())
- : VPDef(SC), VPUser(Operands), DL(DL) {}
+ : VPUser(Operands), DL(DL), SubclassID(SC) {}
+
+ virtual ~VPRecipeBase() {
+ for (VPRecipeValue *D : to_vector(DefinedValues)) {
+ assert(
+ D->Def == this &&
+ "all defined VPValues should point to the containing VPRecipeBase");
+ assert(D->getNumUsers() == 0 &&
+ "all defined VPValues should have no more users");
+ delete D;
+ }
+ }
- ~VPRecipeBase() override = default;
+ /// Returns the only VPValue defined by the VPRecipeBase. Can only be called
+ /// for VPRecipeBases with a single defined value.
+ VPValue *getVPSingleValue() {
+ assert(DefinedValues.size() == 1 && "must have exactly one defined value");
+ assert(DefinedValues[0] && "defined value must be non-null");
+ return DefinedValues[0];
+ }
+ const VPValue *getVPSingleValue() const {
+ assert(DefinedValues.size() == 1 && "must have exactly one defined value");
+ assert(DefinedValues[0] && "defined value must be non-null");
+ return DefinedValues[0];
+ }
+
+ /// Returns the VPValue with index \p I defined by the VPRecipeBase.
+ VPValue *getVPValue(unsigned I) {
+ assert(DefinedValues[I] && "defined value must be non-null");
+ return DefinedValues[I];
+ }
+ const VPValue *getVPValue(unsigned I) const {
+ assert(DefinedValues[I] && "defined value must be non-null");
+ return DefinedValues[I];
+ }
+
+ /// Returns an ArrayRef of the values defined by the VPRecipeBase.
+ ArrayRef<VPRecipeValue *> definedValues() { return DefinedValues; }
+ /// Returns an ArrayRef of the values defined by the VPRecipeBase.
+ ArrayRef<VPRecipeValue *> definedValues() const { return DefinedValues; }
+
+ /// Returns the number of values defined by the VPRecipeBase.
+ unsigned getNumDefinedValues() const { return DefinedValues.size(); }
+
+ /// \return an ID for the concrete type of this object.
+ /// This is used to implement the classof checks. This should not be used
+ /// for any other purpose, as the values may change as LLVM evolves.
+ unsigned getVPRecipeID() const { return SubclassID; }
/// Clone the current recipe.
virtual VPRecipeBase *clone() = 0;
@@ -451,11 +572,6 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
iplist<VPRecipeBase>::iterator eraseFromParent();
/// Method to support type inquiry through isa, cast, and dyn_cast.
- static inline bool classof(const VPDef *D) {
- // All VPDefs are also VPRecipeBases.
- return true;
- }
-
static inline bool classof(const VPUser *U) { return true; }
/// Returns true if the recipe may have side-effects.
@@ -485,9 +601,12 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
void setDebugLoc(DebugLoc NewDL) { DL = NewDL; }
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
+ /// Dump the recipe to stderr (for debugging).
+ LLVM_ABI_FOR_TEST void dump() const;
+
///...
[truncated]
|
Member
|
@llvm/pr-subscribers-vectorizers Author: Florian Hahn (fhahn) ChangesA separate VDef is not needed any longer, fold i into VPRecipeBase to Depends on #172758. Patch is 117.76 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/174282.diff 16 Files Affected:
diff --git a/llvm/docs/VectorizationPlan.rst b/llvm/docs/VectorizationPlan.rst
index 73e9e6098175a..b8db8cab843cf 100644
--- a/llvm/docs/VectorizationPlan.rst
+++ b/llvm/docs/VectorizationPlan.rst
@@ -186,7 +186,8 @@ The low-level design of VPlan comprises of the following classes.
input IR instructions are referred to as "Ingredients" of the Recipe. A Recipe
may specify how its ingredients are to be transformed to produce the output IR
instructions; e.g., cloned once, replicated multiple times or widened
- according to selected VF.
+ according to selected VF. VPRecipeBase also defines zero, one or multiple
+ VPValues, modeling the fact that recipes can produce multiple results.
:VPValue:
The base of VPlan's def-use relations class hierarchy. When instantiated, it
@@ -197,11 +198,6 @@ The low-level design of VPlan comprises of the following classes.
A VPUser represents an entity that uses a number of VPValues as operands.
VPUser is similar in some aspects to LLVM's User class.
-:VPDef:
- A VPDef represents an entity that defines zero, one or multiple VPValues.
- It is used to model the fact that recipes in VPlan can define multiple
- VPValues.
-
:VPInstruction:
A VPInstruction is a recipe characterized by a single opcode and optional
flags, free of ingredients or other meta-data. VPInstructions also extend
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 430d9469c7698..ddef767f6937c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -296,7 +296,7 @@ class VPBuilder {
/// induction with \p Start and \p Step values, using \p Start + \p Current *
/// \p Step.
VPDerivedIVRecipe *createDerivedIV(InductionDescriptor::InductionKind Kind,
- FPMathOperator *FPBinOp, VPValue *Start,
+ FPMathOperator *FPBinOp, VPIRValue *Start,
VPValue *Current, VPValue *Step,
const Twine &Name = "") {
return tryInsertInstruction(
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 04e08e0349074..4ce686bb6cf3d 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -4110,41 +4110,41 @@ static bool willGenerateVectors(VPlan &Plan, ElementCount VF,
// result. Note that this includes VPInstruction where some opcodes may
// produce a vector, to preserve existing behavior as VPInstructions model
// aspects not directly mapped to existing IR instructions.
- switch (R.getVPDefID()) {
- case VPDef::VPDerivedIVSC:
- case VPDef::VPScalarIVStepsSC:
- case VPDef::VPReplicateSC:
- case VPDef::VPInstructionSC:
- case VPDef::VPCanonicalIVPHISC:
- case VPDef::VPVectorPointerSC:
- case VPDef::VPVectorEndPointerSC:
- case VPDef::VPExpandSCEVSC:
- case VPDef::VPEVLBasedIVPHISC:
- case VPDef::VPPredInstPHISC:
- case VPDef::VPBranchOnMaskSC:
+ switch (R.getVPRecipeID()) {
+ case VPRecipeBase::VPDerivedIVSC:
+ case VPRecipeBase::VPScalarIVStepsSC:
+ case VPRecipeBase::VPReplicateSC:
+ case VPRecipeBase::VPInstructionSC:
+ case VPRecipeBase::VPCanonicalIVPHISC:
+ case VPRecipeBase::VPVectorPointerSC:
+ case VPRecipeBase::VPVectorEndPointerSC:
+ case VPRecipeBase::VPExpandSCEVSC:
+ case VPRecipeBase::VPEVLBasedIVPHISC:
+ case VPRecipeBase::VPPredInstPHISC:
+ case VPRecipeBase::VPBranchOnMaskSC:
continue;
- case VPDef::VPReductionSC:
- case VPDef::VPActiveLaneMaskPHISC:
- case VPDef::VPWidenCallSC:
- case VPDef::VPWidenCanonicalIVSC:
- case VPDef::VPWidenCastSC:
- case VPDef::VPWidenGEPSC:
- case VPDef::VPWidenIntrinsicSC:
- case VPDef::VPWidenSC:
- case VPDef::VPWidenSelectSC:
- case VPDef::VPBlendSC:
- case VPDef::VPFirstOrderRecurrencePHISC:
- case VPDef::VPHistogramSC:
- case VPDef::VPWidenPHISC:
- case VPDef::VPWidenIntOrFpInductionSC:
- case VPDef::VPWidenPointerInductionSC:
- case VPDef::VPReductionPHISC:
- case VPDef::VPInterleaveEVLSC:
- case VPDef::VPInterleaveSC:
- case VPDef::VPWidenLoadEVLSC:
- case VPDef::VPWidenLoadSC:
- case VPDef::VPWidenStoreEVLSC:
- case VPDef::VPWidenStoreSC:
+ case VPRecipeBase::VPReductionSC:
+ case VPRecipeBase::VPActiveLaneMaskPHISC:
+ case VPRecipeBase::VPWidenCallSC:
+ case VPRecipeBase::VPWidenCanonicalIVSC:
+ case VPRecipeBase::VPWidenCastSC:
+ case VPRecipeBase::VPWidenGEPSC:
+ case VPRecipeBase::VPWidenIntrinsicSC:
+ case VPRecipeBase::VPWidenSC:
+ case VPRecipeBase::VPWidenSelectSC:
+ case VPRecipeBase::VPBlendSC:
+ case VPRecipeBase::VPFirstOrderRecurrencePHISC:
+ case VPRecipeBase::VPHistogramSC:
+ case VPRecipeBase::VPWidenPHISC:
+ case VPRecipeBase::VPWidenIntOrFpInductionSC:
+ case VPRecipeBase::VPWidenPointerInductionSC:
+ case VPRecipeBase::VPReductionPHISC:
+ case VPRecipeBase::VPInterleaveEVLSC:
+ case VPRecipeBase::VPInterleaveSC:
+ case VPRecipeBase::VPWidenLoadEVLSC:
+ case VPRecipeBase::VPWidenLoadSC:
+ case VPRecipeBase::VPWidenStoreEVLSC:
+ case VPRecipeBase::VPWidenStoreSC:
break;
default:
llvm_unreachable("unhandled recipe");
@@ -7424,11 +7424,12 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// making any changes to the CFG.
DenseMap<const SCEV *, Value *> ExpandedSCEVs =
VPlanTransforms::expandSCEVs(BestVPlan, *PSE.getSE());
- if (!ILV.getTripCount())
+ if (!ILV.getTripCount()) {
ILV.setTripCount(BestVPlan.getTripCount()->getLiveInIRValue());
- else
+ } else {
assert(VectorizingEpilogue && "should only re-use the existing trip "
"count during epilogue vectorization");
+ }
// Perform the actual loop transformation.
VPTransformState State(&TTI, BestVF, LI, DT, ILV.AC, ILV.Builder, &BestVPlan,
@@ -7747,7 +7748,7 @@ VPRecipeBuilder::tryToOptimizeInductionTruncate(VPInstruction *VPI,
auto *WidenIV = cast<VPWidenIntOrFpInductionRecipe>(
VPI->getOperand(0)->getDefiningRecipe());
PHINode *Phi = WidenIV->getPHINode();
- VPValue *Start = WidenIV->getStartValue();
+ VPIRValue *Start = WidenIV->getStartValue();
const InductionDescriptor &IndDesc = WidenIV->getInductionDescriptor();
// It is always safe to copy over the NoWrap and FastMath flags. In
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 6dd250355ac34..f25e76b4015e8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -91,18 +91,6 @@ Value *VPLane::getAsRuntimeExpr(IRBuilderBase &Builder,
llvm_unreachable("Unknown lane kind");
}
-VPValue::VPValue(const unsigned char SC, Value *UV, VPDef *Def)
- : SubclassID(SC), UnderlyingVal(UV), Def(Def) {
- if (Def)
- Def->addDefinedValue(this);
-}
-
-VPValue::~VPValue() {
- assert(Users.empty() && "trying to delete a VPValue with remaining users");
- if (VPDef *Def = getDefiningRecipe())
- Def->removeDefinedValue(this);
-}
-
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
void VPValue::print(raw_ostream &OS, VPSlotTracker &SlotTracker) const {
if (const VPRecipeBase *R = getDefiningRecipe())
@@ -119,21 +107,43 @@ void VPValue::dump() const {
dbgs() << "\n";
}
-void VPDef::dump() const {
- const VPRecipeBase *Instr = dyn_cast_or_null<VPRecipeBase>(this);
- VPSlotTracker SlotTracker(
- (Instr && Instr->getParent()) ? Instr->getParent()->getPlan() : nullptr);
+void VPRecipeBase::dump() const {
+ VPSlotTracker SlotTracker(getParent() ? getParent()->getPlan() : nullptr);
print(dbgs(), "", SlotTracker);
dbgs() << "\n";
}
#endif
VPRecipeBase *VPValue::getDefiningRecipe() {
- return cast_or_null<VPRecipeBase>(Def);
+ auto *DefValue = dyn_cast<VPRecipeValue>(this);
+ if (!DefValue)
+ return nullptr;
+ return DefValue->Def;
}
const VPRecipeBase *VPValue::getDefiningRecipe() const {
- return cast_or_null<VPRecipeBase>(Def);
+ auto *DefValue = dyn_cast<VPRecipeValue>(this);
+ if (!DefValue)
+ return nullptr;
+ return DefValue->Def;
+}
+
+Value *VPValue::getLiveInIRValue() const {
+ return cast<VPIRValue>(this)->getValue();
+}
+
+Type *VPIRValue::getType() const { return getUnderlyingValue()->getType(); }
+
+VPRecipeValue::VPRecipeValue(VPRecipeBase *Def, Value *UV)
+ : VPValue(VPVRecipeValueSC, UV), Def(Def) {
+ assert(Def && "VPRecipeValue requires a defining recipe");
+ Def->addDefinedValue(this);
+}
+
+VPRecipeValue::~VPRecipeValue() {
+ assert(Users.empty() &&
+ "trying to delete a VPRecipeValue with remaining users");
+ Def->removeDefinedValue(this);
}
// Get the top-most entry block of \p Start. This is the entry block of the
@@ -229,8 +239,8 @@ VPTransformState::VPTransformState(const TargetTransformInfo *TTI,
CurrentParentLoop(CurrentParentLoop), TypeAnalysis(*Plan), VPDT(*Plan) {}
Value *VPTransformState::get(const VPValue *Def, const VPLane &Lane) {
- if (Def->isLiveIn())
- return Def->getLiveInIRValue();
+ if (isa<VPIRValue, VPSymbolicValue>(Def))
+ return Def->getUnderlyingValue();
if (hasScalarValue(Def, Lane))
return Data.VPV2Scalars[Def][Lane.mapToCacheIndex(VF)];
@@ -262,8 +272,8 @@ Value *VPTransformState::get(const VPValue *Def, const VPLane &Lane) {
Value *VPTransformState::get(const VPValue *Def, bool NeedsScalar) {
if (NeedsScalar) {
- assert((VF.isScalar() || Def->isLiveIn() || hasVectorValue(Def) ||
- !vputils::onlyFirstLaneUsed(Def) ||
+ assert((VF.isScalar() || isa<VPIRValue, VPSymbolicValue>(Def) ||
+ hasVectorValue(Def) || !vputils::onlyFirstLaneUsed(Def) ||
(hasScalarValue(Def, VPLane(0)) &&
Data.VPV2Scalars[Def].size() == 1)) &&
"Trying to access a single scalar per part but has multiple scalars "
@@ -284,7 +294,6 @@ Value *VPTransformState::get(const VPValue *Def, bool NeedsScalar) {
};
if (!hasScalarValue(Def, {0})) {
- assert(Def->isLiveIn() && "expected a live-in");
Value *IRV = Def->getLiveInIRValue();
Value *B = GetBroadcastInstrs(IRV);
set(Def, B);
@@ -866,7 +875,7 @@ VPlan::VPlan(Loop *L) {
}
VPlan::~VPlan() {
- VPValue DummyValue;
+ VPSymbolicValue DummyValue;
for (auto *VPB : CreatedBlocks) {
if (auto *VPBB = dyn_cast<VPBasicBlock>(VPB)) {
@@ -1053,7 +1062,7 @@ void VPlan::printLiveIns(raw_ostream &O) const {
O << "\n";
if (TripCount) {
- if (TripCount->isLiveIn())
+ if (isa<VPIRValue>(TripCount))
O << "Live-in ";
TripCount->printAsOperand(O, SlotTracker);
O << " = original trip-count";
@@ -1169,20 +1178,17 @@ VPlan *VPlan::duplicate() {
// Create VPlan, clone live-ins and remap operands in the cloned blocks.
auto *NewPlan = new VPlan(cast<VPBasicBlock>(NewEntry), NewScalarHeader);
DenseMap<VPValue *, VPValue *> Old2NewVPValues;
- for (VPValue *OldLiveIn : getLiveIns()) {
- Old2NewVPValues[OldLiveIn] =
- NewPlan->getOrAddLiveIn(OldLiveIn->getLiveInIRValue());
- }
+ for (VPIRValue *OldLiveIn : getLiveIns())
+ Old2NewVPValues[OldLiveIn] = NewPlan->getOrAddLiveIn(OldLiveIn->getValue());
Old2NewVPValues[&VectorTripCount] = &NewPlan->VectorTripCount;
Old2NewVPValues[&VF] = &NewPlan->VF;
Old2NewVPValues[&VFxUF] = &NewPlan->VFxUF;
if (BackedgeTakenCount) {
- NewPlan->BackedgeTakenCount = new VPValue();
+ NewPlan->BackedgeTakenCount = new VPSymbolicValue();
Old2NewVPValues[BackedgeTakenCount] = NewPlan->BackedgeTakenCount;
}
- if (TripCount && TripCount->isLiveIn())
- Old2NewVPValues[TripCount] =
- NewPlan->getOrAddLiveIn(TripCount->getLiveInIRValue());
+ if (auto *LI = dyn_cast_or_null<VPIRValue>(TripCount))
+ Old2NewVPValues[LI] = NewPlan->getOrAddLiveIn(LI->getValue());
// else NewTripCount will be created and inserted into Old2NewVPValues when
// TripCount is cloned. In any case NewPlan->TripCount is updated below.
@@ -1450,7 +1456,7 @@ void VPSlotTracker::assignName(const VPValue *V) {
const auto &[A, _] = VPValue2Name.try_emplace(V, BaseName);
// Integer or FP constants with different types will result in he same string
// due to stripping types.
- if (V->isLiveIn() && isa<ConstantInt, ConstantFP>(UV))
+ if (isa<VPIRValue>(V) && isa<ConstantInt, ConstantFP>(UV))
return;
// If it is already used by C > 0 other VPValues, increase the version counter
@@ -1727,10 +1733,10 @@ bool llvm::canConstantBeExtended(const APInt *C, Type *NarrowType,
TargetTransformInfo::OperandValueInfo
VPCostContext::getOperandInfo(VPValue *V) const {
- if (!V->isLiveIn())
- return {};
+ if (auto *LI = dyn_cast<VPIRValue>(V))
+ return TTI::getOperandInfo(LI->getValue());
- return TTI::getOperandInfo(V->getLiveInIRValue());
+ return {};
}
InstructionCost VPCostContext::getScalarizationOverhead(
@@ -1758,7 +1764,7 @@ InstructionCost VPCostContext::getScalarizationOverhead(
SmallPtrSet<const VPValue *, 4> UniqueOperands;
SmallVector<Type *> Tys;
for (auto *Op : Operands) {
- if (Op->isLiveIn() ||
+ if (isa<VPIRValue>(Op) ||
(!AlwaysIncludeReplicatingR &&
isa<VPReplicateRecipe, VPPredInstPHIRecipe>(Op)) ||
(isa<VPReplicateRecipe>(Op) &&
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index e2dfc4678c6d0..a59bf2c6b444d 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -377,16 +377,16 @@ class LLVM_ABI_FOR_TEST VPBlockBase {
};
/// VPRecipeBase is a base class modeling a sequence of one or more output IR
-/// instructions. VPRecipeBase owns the VPValues it defines through VPDef
-/// and is responsible for deleting its defined values. Single-value
-/// recipes must inherit from VPSingleDef instead of inheriting from both
-/// VPRecipeBase and VPValue separately.
+/// instructions. VPRecipeBase owns the VPValues it defines and is responsible
+/// for deleting its defined values. Single-value recipes must inherit from
+/// VPSingleDefRecipe instead of inheriting from both VPRecipeBase and VPValue
+/// separately.
class LLVM_ABI_FOR_TEST VPRecipeBase
: public ilist_node_with_parent<VPRecipeBase, VPBasicBlock>,
- public VPDef,
public VPUser {
friend VPBasicBlock;
friend class VPBlockUtils;
+ friend class VPRecipeValue;
/// Each VPRecipe belongs to a single VPBasicBlock.
VPBasicBlock *Parent = nullptr;
@@ -394,12 +394,133 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
/// The debug location for the recipe.
DebugLoc DL;
+ /// Subclass identifier (for isa/dyn_cast).
+ const unsigned char SubclassID;
+
+ /// The VPValues defined by this VPRecipeBase.
+ TinyPtrVector<VPRecipeValue *> DefinedValues;
+
+ /// Add \p V as a defined value by this VPRecipeBase.
+ void addDefinedValue(VPRecipeValue *V) {
+ assert(V->Def == this &&
+ "can only add VPValue already linked with this VPRecipeBase");
+ DefinedValues.push_back(V);
+ }
+
+ /// Remove \p V from the values defined by this VPRecipeBase. \p V must be a
+ /// defined value of this VPRecipeBase.
+ void removeDefinedValue(VPRecipeValue *V) {
+ assert(V->Def == this &&
+ "can only remove VPValue linked with this VPRecipeBase");
+ assert(is_contained(DefinedValues, V) &&
+ "VPValue to remove must be in DefinedValues");
+ llvm::erase(DefinedValues, V);
+ V->Def = nullptr;
+ }
+
public:
+ /// An enumeration for keeping track of the concrete subclass of VPRecipeBase
+ /// that is actually instantiated. Values of this enumeration are kept in the
+ /// SubclassID field of the VPRecipeBase objects. They are used for concrete
+ /// type identification.
+ using VPRecipeTy = enum {
+ VPBranchOnMaskSC,
+ VPDerivedIVSC,
+ VPExpandSCEVSC,
+ VPExpressionSC,
+ VPIRInstructionSC,
+ VPInstructionSC,
+ VPInterleaveEVLSC,
+ VPInterleaveSC,
+ VPReductionEVLSC,
+ VPReductionSC,
+ VPReplicateSC,
+ VPScalarIVStepsSC,
+ VPVectorPointerSC,
+ VPVectorEndPointerSC,
+ VPWidenCallSC,
+ VPWidenCanonicalIVSC,
+ VPWidenCastSC,
+ VPWidenGEPSC,
+ VPWidenIntrinsicSC,
+ VPWidenLoadEVLSC,
+ VPWidenLoadSC,
+ VPWidenStoreEVLSC,
+ VPWidenStoreSC,
+ VPWidenSC,
+ VPWidenSelectSC,
+ VPBlendSC,
+ VPHistogramSC,
+ // START: Phi-like recipes. Need to be kept together.
+ VPWidenPHISC,
+ VPPredInstPHISC,
+ // START: SubclassID for recipes that inherit VPHeaderPHIRecipe.
+ // VPHeaderPHIRecipe need to be kept together.
+ VPCanonicalIVPHISC,
+ VPActiveLaneMaskPHISC,
+ VPEVLBasedIVPHISC,
+ VPFirstOrderRecurrencePHISC,
+ VPWidenIntOrFpInductionSC,
+ VPWidenPointerInductionSC,
+ VPReductionPHISC,
+ // END: SubclassID for recipes that inherit VPHeaderPHIRecipe
+ // END: Phi-like recipes
+ VPFirstPHISC = VPWidenPHISC,
+ VPFirstHeaderPHISC = VPCanonicalIVPHISC,
+ VPLastHeaderPHISC = VPReductionPHISC,
+ VPLastPHISC = VPReductionPHISC,
+ };
+
VPRecipeBase(const unsigned char SC, ArrayRef<VPValue *> Operands,
DebugLoc DL = DebugLoc::getUnknown())
- : VPDef(SC), VPUser(Operands), DL(DL) {}
+ : VPUser(Operands), DL(DL), SubclassID(SC) {}
+
+ virtual ~VPRecipeBase() {
+ for (VPRecipeValue *D : to_vector(DefinedValues)) {
+ assert(
+ D->Def == this &&
+ "all defined VPValues should point to the containing VPRecipeBase");
+ assert(D->getNumUsers() == 0 &&
+ "all defined VPValues should have no more users");
+ delete D;
+ }
+ }
- ~VPRecipeBase() override = default;
+ /// Returns the only VPValue defined by the VPRecipeBase. Can only be called
+ /// for VPRecipeBases with a single defined value.
+ VPValue *getVPSingleValue() {
+ assert(DefinedValues.size() == 1 && "must have exactly one defined value");
+ assert(DefinedValues[0] && "defined value must be non-null");
+ return DefinedValues[0];
+ }
+ const VPValue *getVPSingleValue() const {
+ assert(DefinedValues.size() == 1 && "must have exactly one defined value");
+ assert(DefinedValues[0] && "defined value must be non-null");
+ return DefinedValues[0];
+ }
+
+ /// Returns the VPValue with index \p I defined by the VPRecipeBase.
+ VPValue *getVPValue(unsigned I) {
+ assert(DefinedValues[I] && "defined value must be non-null");
+ return DefinedValues[I];
+ }
+ const VPValue *getVPValue(unsigned I) const {
+ assert(DefinedValues[I] && "defined value must be non-null");
+ return DefinedValues[I];
+ }
+
+ /// Returns an ArrayRef of the values defined by the VPRecipeBase.
+ ArrayRef<VPRecipeValue *> definedValues() { return DefinedValues; }
+ /// Returns an ArrayRef of the values defined by the VPRecipeBase.
+ ArrayRef<VPRecipeValue *> definedValues() const { return DefinedValues; }
+
+ /// Returns the number of values defined by the VPRecipeBase.
+ unsigned getNumDefinedValues() const { return DefinedValues.size(); }
+
+ /// \return an ID for the concrete type of this object.
+ /// This is used to implement the classof checks. This should not be used
+ /// for any other purpose, as the values may change as LLVM evolves.
+ unsigned getVPRecipeID() const { return SubclassID; }
/// Clone the current recipe.
virtual VPRecipeBase *clone() = 0;
@@ -451,11 +572,6 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
iplist<VPRecipeBase>::iterator eraseFromParent();
/// Method to support type inquiry through isa, cast, and dyn_cast.
- static inline bool classof(const VPDef *D) {
- // All VPDefs are also VPRecipeBases.
- return true;
- }
-
static inline bool classof(const VPUser *U) { return true; }
/// Returns true if the recipe may have side-effects.
@@ -485,9 +601,12 @@ class LLVM_ABI_FOR_TEST VPRecipeBase
void setDebugLoc(DebugLoc NewDL) { DL = NewDL; }
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
+ /// Dump the recipe to stderr (for debugging).
+ LLVM_ABI_FOR_TEST void dump() const;
+
///...
[truncated]
|
A separate VDef is not needed any longer, fold i into VPRecipeBase to simplify code and class hierarchy. Depends on llvm#172758.
fhahn
force-pushed
the
vplan-merge-vpdef-vprecipebase
branch
from
918baa6 to
4065d58
Compare
ayalz
reviewed
Jan 8, 2026
| if (!DefValue) | ||
| return nullptr; | ||
| return cast<VPRecipeBase>(DefValue->Def); | ||
| return DefValue->Def; |
Collaborator
There was a problem hiding this comment.
Suggested change
| return DefValue->Def; | |
| return DefValue ? DefValue->Def : nullptr; |
| if (!DefValue) | ||
| return nullptr; | ||
| return cast<VPRecipeBase>(DefValue->Def); | ||
| return DefValue->Def; |
Collaborator
There was a problem hiding this comment.
Suggested change
| return DefValue->Def; | |
| return DefValue ? DefValue->Def : nullptr; |
Comment on lines
12
to
15
| /// VPlan models the following entities: | ||
| /// VPValue VPUser VPDef | ||
| /// | | | ||
| /// VPInstruction | ||
| /// VPValue VPUser VPRecipeBase | ||
| /// | | | | ||
| /// VPInstruction ------+ |
Collaborator
There was a problem hiding this comment.
Somewhat inaccurate, better correct or drop - relying on the documentation in VectorizationPlan.rst
Contributor
Author
There was a problem hiding this comment.
Yes, this deserves updating separately I think
| VPDef *Def; | ||
| friend class VPRecipeBase; | ||
| /// Pointer to the VPRecipeBase that defines this VPValue. | ||
| VPRecipeBase *Def; |
Collaborator
There was a problem hiding this comment.
Would it suffice to only migrate the ID's from VPDef to VPRecipeBase along with updating VPRecipeValue to hold a VPRecipeBase* instead of a VPDef*, to simplify code that uses recipes, while retaining VPDef as a base/sub class of VPRecipeBase in charge of its defined values? Possibly as a gradual step towards absorbing VPDef completely inside VPRecipeBase, if that is still desired.
Contributor
Author
There was a problem hiding this comment.
Yep, restored as much of the original code of VPDef, we just need a helper to VPRecipeValue ot check if Def == this, before VPRecipeBase is defined.
fhahn
added a commit
to fhahn/llvm-project
that referenced
this pull request
Jan 11, 2026
Follow-up to llvm#174282: Introduce a new VPConstantInt overlay for VPIRValue, to make it easier to check and access constant int IR values.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A separate VDef is not needed any longer, fold i into VPRecipeBase to
simplify code and class hierarchy.
Depends on #172758.