[SPIRV] Handle AMDGPU buffers in AMDGCN flavoured SPIR-V#199376
Draft
AlexVlx wants to merge 67 commits into
Draft
[SPIRV] Handle AMDGPU buffers in AMDGCN flavoured SPIR-V#199376AlexVlx wants to merge 67 commits into
AlexVlx wants to merge 67 commits into
Conversation
… HIPSTDPAR runtime component.
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
Co-authored-by: Marcos Maronas <marcos.maronas@intel.com>
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
…pirv_be_staging_13
…v_be_staging_13
…v_be_staging_13
…v_be_staging_13
You can test this locally with the following command:git-clang-format --diff origin/main HEAD --extensions c,cpp,h -- clang/lib/CodeGen/CodeGenModule.cpp clang/test/CodeGen/target-data.c llvm/lib/IR/Type.cpp llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp llvm/lib/Target/SPIRV/SPIRVCallLowering.cpp llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.h llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp llvm/lib/Target/SPIRV/SPIRVISelLowering.h llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp llvm/lib/Target/SPIRV/SPIRVLegalizeZeroSizeArrays.cpp llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.h llvm/lib/Target/SPIRV/SPIRVPreLegalizer.cpp llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp llvm/lib/Target/SPIRV/SPIRVPushConstantAccess.cpp llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp llvm/lib/Target/SPIRV/SPIRVUtils.cpp llvm/lib/Target/SPIRV/SPIRVUtils.h llvm/lib/TargetParser/TargetDataLayout.cpp --diff_from_common_commit
View the diff from clang-format here.diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 3afeb5e4c..1b80a3b62 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -418,7 +418,7 @@ static void checkDataLayoutConsistency(const TargetInfo &Target,
Check("void*",
llvm::PointerType::get(Context,
Target.getTargetAddressSpace(LangAS::Default)),
- Target.PointerAlign);
+ Target.PointerAlign);
if (Target.vectorsAreElementAligned() != DL.vectorsAreElementAligned()) {
llvm::errs() << "Datalayout for target " << Triple.str()
diff --git a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
index 06ab77566..be529c3bf 100644
--- a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
@@ -2109,7 +2109,8 @@ SPIRVGlobalRegistry::getRegClass(SPIRVTypeInst SpvType) const {
inline unsigned getAS(SPIRVTypeInst SpvType, const Triple &TT) {
return storageClassToAddressSpace(
static_cast<SPIRV::StorageClass::StorageClass>(
- SpvType->getOperand(1).getImm()), TT);
+ SpvType->getOperand(1).getImm()),
+ TT);
}
LLT SPIRVGlobalRegistry::getRegType(SPIRVTypeInst SpvType) const {
diff --git a/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp b/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
index 8b91ab5fb..de7fa6cd8 100644
--- a/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
@@ -99,8 +99,7 @@ SPIRVLegalizerInfo::SPIRVLegalizerInfo(const SPIRVSubtarget &ST,
storageClassToAddressSpace(SPIRV::StorageClass::DeviceOnlyINTEL, TT);
unsigned HostOnlyINTELAS =
storageClassToAddressSpace(SPIRV::StorageClass::HostOnlyINTEL, TT);
- unsigned InputAS =
- storageClassToAddressSpace(SPIRV::StorageClass::Input, TT);
+ unsigned InputAS = storageClassToAddressSpace(SPIRV::StorageClass::Input, TT);
unsigned OutputAS =
storageClassToAddressSpace(SPIRV::StorageClass::Output, TT);
unsigned CodeSectionINTELAS =
@@ -115,27 +114,27 @@ SPIRVLegalizerInfo::SPIRVLegalizerInfo(const SPIRVSubtarget &ST,
storageClassToAddressSpace(SPIRV::StorageClass::PushConstant, TT);
const LLT p0 = LLT::pointer(FunctionAS, TM.getPointerSizeInBits(FunctionAS));
- const LLT p1 = LLT::pointer(CrossWorkgroupAS,
- TM.getPointerSizeInBits(CrossWorkgroupAS));
+ const LLT p1 =
+ LLT::pointer(CrossWorkgroupAS, TM.getPointerSizeInBits(CrossWorkgroupAS));
const LLT p2 = LLT::pointer(UniformConstantAS,
TM.getPointerSizeInBits(UniformConstantAS));
- const LLT p3 = LLT::pointer(WorkgroupAS,
- TM.getPointerSizeInBits(WorkgroupAS));
+ const LLT p3 =
+ LLT::pointer(WorkgroupAS, TM.getPointerSizeInBits(WorkgroupAS));
const LLT p4 = LLT::pointer(GenericAS, TM.getPointerSizeInBits(GenericAS));
- const LLT p5 = LLT::pointer(DeviceOnlyINTELAS,
- TM.getPointerSizeInBits(DeviceOnlyINTELAS));
- const LLT p6 = LLT::pointer(HostOnlyINTELAS,
- TM.getPointerSizeInBits(HostOnlyINTELAS));
+ const LLT p5 = LLT::pointer(DeviceOnlyINTELAS,
+ TM.getPointerSizeInBits(DeviceOnlyINTELAS));
+ const LLT p6 =
+ LLT::pointer(HostOnlyINTELAS, TM.getPointerSizeInBits(HostOnlyINTELAS));
const LLT p7 = LLT::pointer(InputAS, TM.getPointerSizeInBits(InputAS));
const LLT p8 = LLT::pointer(OutputAS, TM.getPointerSizeInBits(OutputAS));
const LLT p9 = LLT::pointer(CodeSectionINTELAS,
TM.getPointerSizeInBits(CodeSectionINTELAS));
const LLT p10 = LLT::pointer(PrivateAS, TM.getPointerSizeInBits(PrivateAS));
- const LLT p11 = LLT::pointer(StorageBufferAS,
- TM.getPointerSizeInBits(StorageBufferAS));
+ const LLT p11 =
+ LLT::pointer(StorageBufferAS, TM.getPointerSizeInBits(StorageBufferAS));
const LLT p12 = LLT::pointer(UniformAS, TM.getPointerSizeInBits(UniformAS));
- const LLT p13 = LLT::pointer(PushConstantAS,
- TM.getPointerSizeInBits(PushConstantAS));
+ const LLT p13 =
+ LLT::pointer(PushConstantAS, TM.getPointerSizeInBits(PushConstantAS));
// TODO: remove copy-pasting here by using concatenation in some way.
auto allPtrsScalarsAndVectors = {
diff --git a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
index c20394d37..0e58a77ca 100644
--- a/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVPrepareFunctions.cpp
@@ -821,11 +821,10 @@ static inline bool isTargetSpecificASCast(unsigned SrcAS, unsigned DstAS,
storageClassToAddressSpace(SPIRV::StorageClass::UniformConstant, TT);
static const std::pair<unsigned, unsigned> Casts[] = {
- {GenericAS, BufferFatPointerAS}, {GenericAS, BufferResourceAS},
- {BufferFatPointerAS, GenericAS}, {BufferFatPointerAS, BufferResourceAS},
- {BufferResourceAS, GenericAS}, {BufferResourceAS, BufferFatPointerAS},
- {GenericAS, UniformConstAS}, {UniformConstAS, GenericAS}
- };
+ {GenericAS, BufferFatPointerAS}, {GenericAS, BufferResourceAS},
+ {BufferFatPointerAS, GenericAS}, {BufferFatPointerAS, BufferResourceAS},
+ {BufferResourceAS, GenericAS}, {BufferResourceAS, BufferFatPointerAS},
+ {GenericAS, UniformConstAS}, {UniformConstAS, GenericAS}};
return find(Casts, std::make_pair(SrcAS, DstAS));
}
@@ -858,9 +857,8 @@ static bool substituteInvalidAddrSpaceCasts(Function *F) {
auto &ASC = I.get();
B.SetInsertPoint(&ASC);
- CallInst *TASC = B.CreateIntrinsic(ASC.getType(),
- Intrinsic::spv_opaque_ptr_cast,
- {ASC.getOperand(0)});
+ CallInst *TASC = B.CreateIntrinsic(
+ ASC.getType(), Intrinsic::spv_opaque_ptr_cast, {ASC.getOperand(0)});
ASC.replaceAllUsesWith(TASC);
ASC.eraseFromParent();
diff --git a/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp b/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
index f80d02c0d..00838079a 100644
--- a/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVSubtarget.cpp
@@ -191,8 +191,8 @@ void SPIRVSubtarget::setEnv(SPIRVEnvType E) {
// Reinitialize Env-dependent state aka ExtInstSet and legalizer info.
initAvailableExtInstSets();
- Legalizer = std::make_unique<SPIRVLegalizerInfo>(*this,
- TLInfo.getTargetMachine());
+ Legalizer =
+ std::make_unique<SPIRVLegalizerInfo>(*this, TLInfo.getTargetMachine());
}
void SPIRVSubtarget::resolveEnvFromModule(const Module &M) {
diff --git a/llvm/lib/Target/SPIRV/SPIRVUtils.h b/llvm/lib/Target/SPIRV/SPIRVUtils.h
index c5c3be94c..ff728566f 100644
--- a/llvm/lib/Target/SPIRV/SPIRVUtils.h
+++ b/llvm/lib/Target/SPIRV/SPIRVUtils.h
@@ -248,7 +248,7 @@ constexpr bool isGenericCastablePtr(SPIRV::StorageClass::StorageClass SC) {
// to allow for different OpenCL vs Vulkan handling.
constexpr unsigned
storageClassToAddressSpace(SPIRV::StorageClass::StorageClass SC,
- const Triple& TT) {
+ const Triple &TT) {
switch (SC) {
case SPIRV::StorageClass::Function:
return 0;
diff --git a/llvm/lib/TargetParser/TargetDataLayout.cpp b/llvm/lib/TargetParser/TargetDataLayout.cpp
index 22fec2f2e..d6e4178ea 100644
--- a/llvm/lib/TargetParser/TargetDataLayout.cpp
+++ b/llvm/lib/TargetParser/TargetDataLayout.cpp
@@ -489,10 +489,10 @@ static std::string computeSPIRVDataLayout(const Triple &TT) {
if (TT.getVendor() == Triple::VendorType::AMD &&
TT.getOS() == Triple::OSType::AMDHSA) {
auto DL = computeAMDDataLayout(TT);
- DL.replace(DL.find("p:64:64"), 7, "p:32:32"); // AS0 is Function
+ DL.replace(DL.find("p:64:64"), 7, "p:32:32"); // AS0 is Function
DL.replace(DL.find("p2:32:32"), 8, "p2:64:64"); // AS2 is UniformConstant
- DL.replace(DL.find("A5"), 2, "A0"); // AllocaAS is Function
- DL.insert(DL.find_last_of('-') + 1, "P4-"); // ProgramAS is Generic
+ DL.replace(DL.find("A5"), 2, "A0"); // AllocaAS is Function
+ DL.insert(DL.find_last_of('-') + 1, "P4-"); // ProgramAS is Generic
return DL;
}
if (TT.getVendor() == Triple::VendorType::Intel)
|
AlexVlx
requested review from
MrSidims,
aobolensk,
jhuber6,
jmmartinez,
maarquitos14,
michalpaszkowski and
rampitec
|
@llvm/pr-subscribers-backend-spir-v Author: Alex Voicu (AlexVlx) ChangesThis is a bit of a chonk because it includes a bunch of related / necessary changes (could have been a stack, but my stack-fu is weak). I'll first merge in the various bits and bobs and then un-draft it, however it seems that getting some early discussion would be very helpful. As to the change itself: AMDGPU has fairly elaborate infrastructure around buffer ops, which is closely tied to bespoke address spaces 7 & 8 + non-trivial lowering of operations on pointers to said ASes. We want to retain the AS info in SPIR-V, so that we can work the magic when we finalise to AMDGPU. The challenge is that in SPIR-V, 7 and 8 correspond to the
Whilst this is conceptually pretty straightforward, it turns out that a bunch of plumbing was needed, since AS7 and AS8 have somewhat oddly layouts (e.g. the pointer size is 160 and 256 bits respectively). Patch is 81.17 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/199376.diff 27 Files Affected:
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 236738e9975d3..3afeb5e4cf7c5 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -415,7 +415,10 @@ static void checkDataLayoutConsistency(const TargetInfo &Target,
if (Target.hasIbm128Type())
Check("__ibm128", llvm::Type::getPPC_FP128Ty(Context), Target.Ibm128Align);
- Check("void*", llvm::PointerType::getUnqual(Context), Target.PointerAlign);
+ Check("void*",
+ llvm::PointerType::get(Context,
+ Target.getTargetAddressSpace(LangAS::Default)),
+ Target.PointerAlign);
if (Target.vectorsAreElementAligned() != DL.vectorsAreElementAligned()) {
llvm::errs() << "Datalayout for target " << Triple.str()
diff --git a/clang/test/CodeGen/target-data.c b/clang/test/CodeGen/target-data.c
index 0047e1377a7f8..a73531a7714d6 100644
--- a/clang/test/CodeGen/target-data.c
+++ b/clang/test/CodeGen/target-data.c
@@ -259,7 +259,7 @@
// RUN: %clang_cc1 -triple spirv64-amd-amdhsa -o - -emit-llvm %s | \
// RUN: FileCheck %s -check-prefix=AMDGPUSPIRV64
-// AMDGPUSPIRV64: target datalayout = "e-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-n32:64-S32-G1-P4-A0"
+// AMDGPUSPIRV64: target datalayout = "e-m:e-p:32:32-p1:64:64-p2:64:64-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-p9:192:256:256:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A0-G1-P4-ni:7:8:9"
// RUN: %clang_cc1 -triple spirv-unknown-vulkan -o - -emit-llvm %s | \
// RUN: FileCheck %s -check-prefix=SPIRVVULKAN
diff --git a/llvm/include/llvm/IR/IntrinsicsSPIRV.td b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
index 26996774afa87..d728ed080dca4 100644
--- a/llvm/include/llvm/IR/IntrinsicsSPIRV.td
+++ b/llvm/include/llvm/IR/IntrinsicsSPIRV.td
@@ -354,6 +354,11 @@ def int_spv_rsqrt : DefaultAttrsIntrinsic<[LLVMMatchType<0>], [llvm_anyfloat_ty]
def int_spv_generic_cast_to_ptr_explicit
: DefaultAttrsIntrinsic<[llvm_anyptr_ty], [generic_ptr_ty],
[IntrNoMem, NoUndef<RetIndex>]>;
+ // Encode SPIR-V invalid but potentially target valid casts via intrinsic that
+ // always gets lowered to a function.
+ def int_spv_opaque_ptr_cast
+ : DefaultAttrsIntrinsic<[llvm_anyptr_ty], [llvm_anyptr_ty],
+ [IntrNoMem, NoUndef<RetIndex>]>;
def int_spv_unpackhalf2x16 : DefaultAttrsIntrinsic<[llvm_anyfloat_ty], [llvm_i32_ty], [IntrNoMem]>;
def int_spv_packhalf2x16 : DefaultAttrsIntrinsic<[llvm_anyint_ty], [llvm_anyfloat_ty], [IntrNoMem]>;
diff --git a/llvm/lib/IR/Type.cpp b/llvm/lib/IR/Type.cpp
index 498e78d4b0cc8..4be20db1c0d33 100644
--- a/llvm/lib/IR/Type.cpp
+++ b/llvm/lib/IR/Type.cpp
@@ -1082,6 +1082,10 @@ static TargetTypeInfo getTargetTypeInfo(const TargetExtType *Ty) {
return TargetTypeInfo(
ArrayType::get(Type::getInt8Ty(C), Ty->getIntParameter(0)),
TargetExtType::CanBeGlobal);
+ if (Name == "spirv.$TypedPointerType")
+ return TargetTypeInfo(PointerType::get(C, 4), TargetExtType::HasZeroInit,
+ TargetExtType::CanBeGlobal, TargetExtType::CanBeLocal,
+ TargetExtType::CanBeVectorElement);
if (Name.starts_with("spirv."))
return TargetTypeInfo(PointerType::get(C, 0), TargetExtType::HasZeroInit,
TargetExtType::CanBeGlobal,
diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
index 1febfc21ed3b0..6ef8715783748 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
@@ -510,10 +510,10 @@ static Register buildBuiltinVariableLoad(
SPIRV::LinkageType::Import}) {
Register NewRegister =
MIRBuilder.getMRI()->createVirtualRegister(&SPIRV::pIDRegClass);
- MIRBuilder.getMRI()->setType(
- NewRegister,
- LLT::pointer(storageClassToAddressSpace(SPIRV::StorageClass::Function),
- GR->getPointerSize()));
+ unsigned AS = storageClassToAddressSpace(
+ SPIRV::StorageClass::Function, GR->CurMF->getTarget().getTargetTriple());
+ MIRBuilder.getMRI()->setType(NewRegister,
+ LLT::pointer(AS, GR->getPointerSize(AS)));
SPIRVTypeInst PtrType = GR->getOrCreateSPIRVPointerType(
VariableType, MIRBuilder, SPIRV::StorageClass::Input);
GR->assignSPIRVTypeToVReg(PtrType, NewRegister, MIRBuilder.getMF());
@@ -2929,7 +2929,8 @@ static SPIRVTypeInst
getOrCreateSPIRVDeviceEventPointer(MachineIRBuilder &MIRBuilder,
SPIRVGlobalRegistry *GR) {
LLVMContext &Context = MIRBuilder.getMF().getFunction().getContext();
- unsigned SC1 = storageClassToAddressSpace(SPIRV::StorageClass::Generic);
+ unsigned SC1 = storageClassToAddressSpace(
+ SPIRV::StorageClass::Generic, GR->CurMF->getTarget().getTargetTriple());
Type *PtrType = PointerType::get(Context, SC1);
return GR->getOrCreateSPIRVType(PtrType, MIRBuilder,
SPIRV::AccessQualifier::ReadWrite, true);
@@ -2960,8 +2961,9 @@ static bool buildEnqueueKernel(const SPIRV::IncomingCall *Call,
assert(LocalSizeTy && "Local size type is expected");
const uint64_t LocalSizeNum =
cast<ArrayType>(LocalSizeTy)->getNumElements();
- unsigned SC = storageClassToAddressSpace(SPIRV::StorageClass::Generic);
- const LLT LLType = LLT::pointer(SC, GR->getPointerSize());
+ unsigned SC = storageClassToAddressSpace(
+ SPIRV::StorageClass::Generic, GR->CurMF->getTarget().getTargetTriple());
+ const LLT LLType = LLT::pointer(SC, GR->getPointerSize(SC));
const SPIRVTypeInst PointerSizeTy = GR->getOrCreateSPIRVPointerType(
Int32Ty, MIRBuilder, SPIRV::StorageClass::Function);
for (unsigned I = 0; I < LocalSizeNum; ++I) {
diff --git a/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp b/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
index b6e71c7b76348..48f12faf8db6f 100644
--- a/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
@@ -1638,12 +1638,17 @@ void SPIRVEmitIntrinsics::preprocessUndefs(IRBuilder<> &B) {
// pointer.
void SPIRVEmitIntrinsics::simplifyNullAddrSpaceCasts() {
for (Instruction &I : make_early_inc_range(instructions(CurrF)))
- if (auto *ASC = dyn_cast<AddrSpaceCastInst>(&I))
+ if (auto *ASC = dyn_cast<AddrSpaceCastInst>(&I)) {
if (isa<ConstantPointerNull>(ASC->getPointerOperand())) {
- ASC->replaceAllUsesWith(
- ConstantPointerNull::get(cast<PointerType>(ASC->getType())));
+ if (ASC->getType()->isVectorTy()) {
+ ASC->replaceAllUsesWith(ConstantVector::getNullValue(ASC->getType()));
+ } else {
+ ASC->replaceAllUsesWith(
+ ConstantPointerNull::get(cast<PointerType>(ASC->getType())));
+ }
ASC->eraseFromParent();
}
+ }
}
void SPIRVEmitIntrinsics::preprocessCompositeConstants(IRBuilder<> &B) {
diff --git a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
index 343311fb44475..06ab77566753c 100644
--- a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.cpp
@@ -763,7 +763,7 @@ SPIRVGlobalRegistry::getOrCreateConstNullPtr(MachineIRBuilder &MIRBuilder,
if (Res.isValid())
return Res;
- LLT LLTy = LLT::pointer(AddressSpace, getPointerSize());
+ LLT LLTy = LLT::pointer(AddressSpace, getPointerSize(AddressSpace));
Res = CurMF->getRegInfo().createGenericVirtualRegister(LLTy);
CurMF->getRegInfo().setRegClass(Res, &SPIRV::pIDRegClass);
assignSPIRVTypeToVReg(SpvType, Res, *CurMF);
@@ -863,8 +863,8 @@ Register SPIRVGlobalRegistry::buildGlobalVariable(
// Set to Reg the same type as ResVReg has.
auto MRI = MIRBuilder.getMRI();
if (Reg != ResVReg) {
- LLT RegLLTy =
- LLT::pointer(MRI->getType(ResVReg).getAddressSpace(), getPointerSize());
+ unsigned AS = MRI->getType(ResVReg).getAddressSpace();
+ LLT RegLLTy = LLT::pointer(AS, getPointerSize(AS));
MRI->setType(Reg, RegLLTy);
assignSPIRVTypeToVReg(BaseType, Reg, MIRBuilder.getMF());
} else {
@@ -2033,7 +2033,8 @@ SPIRVTypeInst SPIRVGlobalRegistry::getOrCreateSPIRVPointerTypeInternal(
SPIRVTypeInst BaseType, MachineIRBuilder &MIRBuilder,
SPIRV::StorageClass::StorageClass SC) {
const Type *PointerElementType = getTypeForSPIRVType(BaseType);
- unsigned AddressSpace = storageClassToAddressSpace(SC);
+ unsigned AddressSpace =
+ storageClassToAddressSpace(SC, CurMF->getTarget().getTargetTriple());
if (const MachineInstr *MI = findMI(PointerElementType, AddressSpace, CurMF))
return MI;
Type *Ty = TypedPointerType::get(const_cast<Type *>(PointerElementType),
@@ -2105,28 +2106,33 @@ SPIRVGlobalRegistry::getRegClass(SPIRVTypeInst SpvType) const {
return &SPIRV::iIDRegClass;
}
-inline unsigned getAS(SPIRVTypeInst SpvType) {
+inline unsigned getAS(SPIRVTypeInst SpvType, const Triple &TT) {
return storageClassToAddressSpace(
static_cast<SPIRV::StorageClass::StorageClass>(
- SpvType->getOperand(1).getImm()));
+ SpvType->getOperand(1).getImm()), TT);
}
LLT SPIRVGlobalRegistry::getRegType(SPIRVTypeInst SpvType) const {
unsigned Opcode = SpvType ? SpvType->getOpcode() : 0;
+ const Triple &TT = CurMF->getTarget().getTargetTriple();
switch (Opcode) {
case SPIRV::OpTypeInt:
case SPIRV::OpTypeFloat:
case SPIRV::OpTypeBool:
return LLT::scalar(getScalarOrVectorBitWidth(SpvType));
- case SPIRV::OpTypePointer:
- return LLT::pointer(getAS(SpvType), getPointerSize());
+ case SPIRV::OpTypePointer: {
+ unsigned AS = getAS(SpvType, TT);
+ return LLT::pointer(AS, getPointerSize(AS));
+ }
case SPIRV::OpTypeVector: {
SPIRVTypeInst ElemType = getScalarOrVectorComponentType(SpvType);
LLT ET;
switch (ElemType ? ElemType->getOpcode() : 0) {
- case SPIRV::OpTypePointer:
- ET = LLT::pointer(getAS(ElemType), getPointerSize());
+ case SPIRV::OpTypePointer: {
+ unsigned AS = getAS(ElemType, TT);
+ ET = LLT::pointer(AS, getPointerSize(AS));
break;
+ }
case SPIRV::OpTypeInt:
case SPIRV::OpTypeFloat:
case SPIRV::OpTypeBool:
diff --git a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.h b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.h
index f05bdc2cc9861..427aab376d69d 100644
--- a/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.h
+++ b/llvm/lib/Target/SPIRV/SPIRVGlobalRegistry.h
@@ -418,8 +418,8 @@ class SPIRVGlobalRegistry : public SPIRVIRMapping {
getPointerStorageClass(SPIRVTypeInst Type) const;
// Return the number of bits SPIR-V pointers and size_t variables require.
- unsigned getPointerSize() const {
- return DL.getPointerSizeInBits(/* AS = */ 0);
+ unsigned getPointerSize(unsigned AS = 0) const {
+ return DL.getPointerSizeInBits(AS);
}
// Returns true if two types are defined and are compatible in a sense of
diff --git a/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp b/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
index eb16c9a314a23..c5499e9d6c1ea 100644
--- a/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVISelLowering.cpp
@@ -21,6 +21,7 @@
#include "llvm/CodeGen/TargetLowering.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicsSPIRV.h"
+#include "llvm/Support/AMDGPUAddrSpace.h"
#define DEBUG_TYPE "spirv-lower"
@@ -102,6 +103,12 @@ MVT SPIRVTargetLowering::getRegisterTypeForCallingConv(LLVMContext &Context,
else if (VT.getVectorElementType() == MVT::i8)
return MVT::v4i8;
}
+ // if (STI.getTargetTriple().getVendor() == Triple::VendorType::AMD) {
+ // if (VT.isVector() &&
+ // VT.getVectorElementType().isExtended())
+ // return MVT::v8i32;
+ // }
+
return getRegisterType(Context, VT);
}
@@ -684,3 +691,33 @@ SPIRVTargetLowering::shouldCastAtomicRMWIInIR(AtomicRMWInst *RMWI) const {
// SPIR-V only supports atomic exchange for integer and floating-point types.
return AtomicExpansionKind::None;
}
+
+// These are specific to AMDGCN flavoured SPIR-V, and mirror the AMDGPU
+// implementation; they are required because buffer operations and the machinery
+// around them are somewhat special.
+MVT SPIRVTargetLowering::getPointerTy(const DataLayout &DL, unsigned AS) const {
+ if (STI.getTargetTriple().getVendor() != Triple::VendorType::AMD)
+ return TargetLowering::getPointerTy(DL, AS);
+
+ if (AMDGPUAS::BUFFER_FAT_POINTER == AS && DL.getPointerSizeInBits(AS) == 160)
+ return MVT::amdgpuBufferFatPointer;
+ if (AMDGPUAS::BUFFER_STRIDED_POINTER == AS &&
+ DL.getPointerSizeInBits(AS) == 192)
+ return MVT::amdgpuBufferStridedPointer;
+
+ return TargetLowering::getPointerTy(DL, AS);
+}
+
+MVT SPIRVTargetLowering::getPointerMemTy(const DataLayout &DL,
+ unsigned AS) const {
+ if (STI.getTargetTriple().getVendor() != Triple::VendorType::AMD)
+ return TargetLowering::getPointerMemTy(DL, AS);
+
+ if ((AMDGPUAS::BUFFER_FAT_POINTER == AS &&
+ DL.getPointerSizeInBits(AS) == 160) ||
+ (AMDGPUAS::BUFFER_STRIDED_POINTER == AS &&
+ DL.getPointerSizeInBits(AS) == 192))
+ return MVT::v8i32;
+
+ return TargetLowering::getPointerMemTy(DL, AS);
+}
diff --git a/llvm/lib/Target/SPIRV/SPIRVISelLowering.h b/llvm/lib/Target/SPIRV/SPIRVISelLowering.h
index 6af0bbcc9818d..60710610c6c28 100644
--- a/llvm/lib/Target/SPIRV/SPIRVISelLowering.h
+++ b/llvm/lib/Target/SPIRV/SPIRVISelLowering.h
@@ -82,6 +82,9 @@ class SPIRVTargetLowering : public TargetLowering {
bool shouldIssueAtomicLoadForAtomicEmulationLoop() const override {
return false;
}
+
+ MVT getPointerTy(const DataLayout &DL, unsigned AS) const override;
+ MVT getPointerMemTy(const DataLayout &DL, unsigned AS) const override;
};
} // namespace llvm
diff --git a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
index 02b41b9cbd64c..df03b0cf64bf8 100644
--- a/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVInstructionSelector.cpp
@@ -2565,9 +2565,9 @@ SPIRVInstructionSelector::buildConstGenericPtr(MachineInstr &I, Register SrcPtr,
SPIRVTypeInst GenericPtrTy =
GR.changePointerStorageClass(SrcPtrTy, SPIRV::StorageClass::Generic, I);
Register Tmp = MRI->createVirtualRegister(&SPIRV::pIDRegClass);
- MRI->setType(Tmp, LLT::pointer(storageClassToAddressSpace(
- SPIRV::StorageClass::Generic),
- GR.getPointerSize()));
+ unsigned AS = storageClassToAddressSpace(SPIRV::StorageClass::Generic,
+ STI.getTargetTriple());
+ MRI->setType(Tmp, LLT::pointer(AS, GR.getPointerSize(AS)));
MachineFunction *MF = I.getParent()->getParent();
GR.assignSPIRVTypeToVReg(GenericPtrTy, Tmp, *MF);
MachineInstrBuilder MIB = buildSpecConstantOp(
@@ -2626,6 +2626,25 @@ bool SPIRVInstructionSelector::selectAddrSpaceCast(Register ResVReg,
I, ResVReg, MIB->getOperand(0).getReg(), getUcharPtrTypeReg(I, DstSC),
static_cast<uint32_t>(SPIRV::Opcode::GenericCastToPtr))
.constrainAllUses(TII, TRI, RBI);
+ } else if (STI.getTargetTriple().getVendor() == Triple::VendorType::AMD &&
+ isUSMStorageClass(SrcSC) && isUSMStorageClass(DstSC)) {
+ // For AMDGCN flavoured SPIR-V we repurpose DeviceOnlyINTEL and
+ // HostOnlyINTEL (which we do not use) to represent Buffer specific
+ // storage. Pointers to Buffer are essentially opaque handles, and have
+ // non trivial lowering, so we only care about retaining information about
+ // in their nature in SPIR-V, as they can only be handled properly during
+ // finalisation. Since they are inter-castable, and there is no valid
+ // SPIR-V cast between DeviceOnlyINTEL and HostOnlyIntel, we use a PtrToU
+ // followed by an UToPtr pattern to encode the cast, pattern that we match
+ // and retrieve at finalisation.
+ auto I64Ty = GR.getOrCreateSPIRVIntegerType(64, I, TII);
+ Register Tmp = createVirtualRegister(I64Ty, &GR, MRI, MRI->getMF());
+ auto MIB = buildSpecConstantOp(I, Tmp, SrcPtr, GR.getSPIRVTypeID(I64Ty),
+ SPIRV::Opcode::ConvertPtrToU);
+ MIB.constrainAllUses(TII, TRI, RBI);
+ buildSpecConstantOp(I, ResVReg, Tmp, getUcharPtrTypeReg(I, DstSC),
+ SPIRV::Opcode::ConvertUToPtr)
+ .constrainAllUses(TII, TRI, RBI);
}
return true;
}
@@ -6809,7 +6828,8 @@ bool SPIRVInstructionSelector::selectModf(Register ResVReg,
MIRBuilder.getMRI()->createVirtualRegister(&SPIRV::iIDRegClass);
MIRBuilder.getMRI()->setType(
PtrTyReg,
- LLT::pointer(storageClassToAddressSpace(SPIRV::StorageClass::Function),
+ LLT::pointer(storageClassToAddressSpace(SPIRV::StorageClass::Function,
+ STI.getTargetTriple()),
GR.getPointerSize()));
// Assign SPIR-V type of the pointer type of the alloca variable to the
@@ -6924,10 +6944,11 @@ bool SPIRVInstructionSelector::loadBuiltinInputID(
// Create new register for the input ID builtin variable.
Register NewRegister =
MIRBuilder.getMRI()->createVirtualRegister(GR.getRegClass(PtrType));
+ unsigned AS = storageClassToAddressSpace(SPIRV::StorageClass::Input,
+ STI.getTargetTriple());
MIRBuilder.getMRI()->setType(
NewRegister,
- LLT::pointer(storageClassToAddressSpace(SPIRV::StorageClass::Input),
- GR.getPointerSize()));
+ LLT::pointer(AS, GR.getPointerSize(AS)));
GR.assignSPIRVTypeToVReg(PtrType, NewRegister, MIRBuilder.getMF());
// Build global variable with the necessary decorations for the input ID
diff --git a/llvm/lib/Target/SPIRV/SPIRVLegalizeZeroSizeArrays.cpp b/llvm/lib/Target/SPIRV/SPIRVLegalizeZeroSizeArrays.cpp
index 09697b59c374a..0991ed14e7df0 100644
--- a/llvm/lib/Target/SPIRV/SPIRVLegalizeZeroSizeArrays.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVLegalizeZeroSizeArrays.cpp
@@ -119,7 +119,8 @@ Type *SPIRVLegalizeZeroSizeArraysImpl::legalizeType(Type *Ty) {
if (isa<ArrayType>(Ty)) {
LegalizedTy = PointerType::get(
Ty->getContext(),
- storageClassToAddressSpace(SPIRV::StorageClass::Generic));
+ storageClassToAddressSpace(SPIRV::StorageClass::Generic,
+ TM.getTargetTriple()));
} else if (StructType *StructTy = dyn_cast<StructType>(Ty)) {
SmallVector<Type *, 8> ElemTypes;
diff --git a/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp b/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
index dd48aa3ea2f0f..bfd937d47b6f1 100644
--- a/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVLegalizerInfo.cpp
@@ -14,6 +14,7 @@
#include "SPIRV.h"
#include "SPIRVGlobalRegistry.h"
#include "SPIRVSubtarget.h"
+#include "SPIRVTargetMachine.h"
#include "SPIRVUtils.h"
#include "llvm/CodeGen/GlobalISel/GenericMachineInstrs.h"
#include "llvm/CodeGen/GlobalISel/LegalizerHelper.h"
@@ -38,7 +39,8 @@ LegalityPredicate typeOfExtendedScalars(unsigned TypeIdx, bool IsExtendedInts) {
};
}
-SPIRVLegalizerInfo::SPIRVLegalizerInfo(const SPIRVSubtarget &ST) {
+SPIRVLegalizerInfo::SPIRVLegalizerInfo(const SPIRVSubtarget &ST,
+ const TargetMachine &TM) {
using namespace TargetOpcode;
this->ST = &ST;
@@ -81,23 +83,24 @@ SPIRVLegalizerInfo::SPIRVLegalizerInfo(const SPIRVSubtarget &ST) {
const LLT v2s8 = LLT::fixed_vector(2, 8);
const LLT v2s1 = LLT::fixed_vector(2, 1);
- const unsigned PSize = ST.getPointerSize();
- const LLT p0 = LLT::pointer(0, PSize); // Function
- const LLT p1 = LLT::pointer(1, PSize); // CrossWorkgroup
- const LLT p2 = LLT::pointer(2, PSize); // UniformConstant
- const LLT p3 = LLT::pointer(3, PSize); // Workgroup
- const LLT p4 = LLT::pointer(4, PSize); // Generic
- const LLT p5 =
- LLT::pointer(5, PSize); // Input, SPV_INTEL_usm_storage_classes (Device)
- const LLT p6 = LLT::pointer(6, PSize); // SPV_INTEL_usm_storage_classes (Host)
- const LLT p7 = LLT::pointer(7, PSize); // Input
- const LLT p8 = LL...
[truncated]
|
| : SPIRVGenSubtargetInfo(TT, CPU, /*TuneCPU=*/CPU, FS), | ||
| PointerSize(TM.getPointerSizeInBits(/* AS= */ 0)), | ||
| PointerSize(TM.getPointerSizeInBits( | ||
| /* AS= */ storageClassToAddressSpace(SPIRV::StorageClass::Generic, |
Contributor
There was a problem hiding this comment.
This switch from AS0 to AS4 is not mentioned/discussed in the PR description.
Contributor
|
An obvious problem with the quick path of appropriation of DeviceOnlyINTEL/HostOnlyINTEL is that Storage Class stops having a semantic meaning by itself, and we are able to interpret it only in the context of a vendor. The standard extension mechanism guarantees that a SPIR-V module can be interpreted in the context of its OpExtension instructions, denoting extensions that added semantics referred by the module. Vendor info as a source of truth is a shift from "SPIR-V module can be interpreted with the specification and extension docs" premise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a bit of a chonk because it includes a bunch of related / necessary changes (could have been a stack, but my stack-fu is weak). I'll first merge in the various bits and bobs and then un-draft it, however it seems that getting some early discussion would be very helpful.
As to the change itself: AMDGPU has fairly elaborate infrastructure around buffer ops, which is closely tied to bespoke address spaces 7 & 8 + non-trivial lowering of operations on pointers to said ASes. We want to retain the AS info in SPIR-V, so that we can work the magic when we finalise to AMDGPU. The challenge is that in SPIR-V, 7 and 8 correspond to the
Inputand, respectively,Outputstorage which have restrictive semantics (the latter is outright unavailable forKernelSPIR-V). What we end up doing is:PtrToUfollowed byUToPtrSpecConstantOpsDeviceOnlyINTELand AS8 toHostOnlyINTEL, which we do not use and do not plan to use at any point.Whilst this is conceptually pretty straightforward, it turns out that a bunch of plumbing was needed, since AS7 and AS8 have somewhat oddly layouts (e.g. the pointer size is 160 and 256 bits respectively).