With D134950, targets get notified when a virtual register is created and/or
cloned. Targets can do the needful with the delegate callback. AMDGPU propagates
the virtual register flags maintained in the target file itself. They are useful
to identify a certain type of machine operands while inserting spill stores and
reloads. Since RegAllocFast spills the physical register itself, there is no way
its virtual register can be mapped back to retrieve the flags. It can be solved
by passing the virtual register as an additional argument. This argument has no
use when the spill interfaces are called during the greedy allocator or even the
PrologEpilogInserter and can pass a null register in such cases.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D138656
If the divisor is repeated at least twice, we will convert the FDIVs to the
calculation of the reciprocal and FMULs.
We perform the transformation only under fast-math mode. FDIVs must have
'arcp' flag.
Differential Revision: https://reviews.llvm.org/D140024
This is a follow up to patch discussion on D139656. As noted there, M2/M4/M8 versions of these instructions don't actually exist, and using them results in overly constrained register allocation.
In that review, we'd talked about moving towards a variant of the instructions which ignored LMUL. I decided to see what happened if we just stopped generating the high LMUL variants, and the results are surprisingly neutral. I only see one minor thing which looks like a real regression among all the churn. I think this is worth doing now to loosen register allocation constraints, and avoid digging our hole around these instructions deeper while thinking about the right model change.
Differential Revision: https://reviews.llvm.org/D140027
The patch also adds expandVPCTPOP in TargetLowering to expand VP_CTPOP nodes.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D139920
This is an alternative patch on a path to D137530.
The basic problem being tackled here is that we need to place a scalar into lane 0 of a vector register before our reduction instructions. Since we only care about lane 0 of the vector, we can use any VL >= 1 provided that the total amount of work performed matches the work performed for a VL=1.
This change does not contain the logic from D137530 to perform the insert at the original VT, and then extract down to LMUL1. That turns out to be a good choice, as discussion in this review has indicated there are issues around LMUL2 and above with our representation of vmv.s.x. We'd also need to be careful with the splat logic for the same reasons.
The only potentially concerning codegen change I spot here is that we stop using a broadcast load (for VL=1) and instead do a scalar load and insert. I think this is probably reasonable; if reviewers disagree, I can investigate using a broadcast load which writes to the undef lanes. If we want to do that, we should do it for VECTOR_INSERT_ELT as well, so that'll end up as it's own patch series.
Differential Revision: https://reviews.llvm.org/D139656
Remove the call to getSExtOrTrunc.
Reduction ISD nodes produce a scalar result and that result is
allowed to be larger than the vector element type due to type
legalization. This is the same rule we allow for EXTRACT_VECTOR_ELT
for the same reason.
We can copy the result type over from the reduction node to
EXTRACT_VECTOR_ELT.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D139757
This doesn't cause any functional change since this is being applied to a insert generated by the same routine. This is mostly about consolidating the logic for vmv.s.x into one place to simplify future changes.
Use the newly introduced helper routine. At the moment, this generates the same code (at this call site!) since LMUL is restricted to LMUL1 or less, and VL is hard coded to 1. In a future patch, I will loosen the second part.
If we're initializing lane 0 of an undef vector, we can optionally write to other lanes of the vector. Doing so may require additional work, so we don't want to e.g. always use a splat. However, since we don't have an immediate form of vmv.s.x it's useful to use a vmv.v.i if the work required is expected to be equal in practice. We restrict this to when LMUL <= 1 to a) prevent doing additional work at higher LMULs, and b) avoid overconstraining the register allocator.
At the moment, the new utility is only used by one case in INSERT_VECTOR_ELT lowering. My expectation is that we will reuse this in a couple other places, but each of those deserve individual review.
This change is inspired by D137530, but is not directly related to it. I vaguely remember we discussed the tradeoffs of using vmv.v.i in another recent review, but couldn't find it.
Differential Revision: https://reviews.llvm.org/D139648
Now with fix to limit added tagged-globals.ll to risc-v platform
--
[RISC-V][HWASAN] Support tagging global variables for RISC-V HWASAN
Reviewed by: luismarques
Differential Revision: https://reviews.llvm.org/D132995
For reductions, we need to put the start value into a source vector. For fractional LMULs, we can perform the operation at the original LMUL. For LMUL > 1, we eventually want to use a scalar insert, but that's outside the scope of this patch.
Differential Revision: https://reviews.llvm.org/D139747
The patch also added function expandVPBITREVERSE to expand ISD::VP_BITREVERSE nodes.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D139697
We can consolidate code and clarify edge case behavior at the same time.
There are two functional differences here.
First, I remove the ResVT handling, and always use the reduction element type. This appears to be dead code. There's no test coverage, and this code doesn't need to account for scalar type legalization anyways.
Second, if the VL happens to be known non-zero, we can avoid passing through start. This is mostly needed to allow reuse of the existing code; I don't consider it interesting as an optimization on it's own.
Differential Revision: https://reviews.llvm.org/D139733
There's several patches in flght which change this code, better to only have one copy.
The VP case is left seperate for the moment as the result value type differs.
I've enabled Zfhmin on 2 basic tests to show this isn't
completely broken.
Reviewed By: monkchiang
Differential Revision: https://reviews.llvm.org/D139562
This reverts commit 7883e5b061.
The original commit was reverted that it didn't update test files after D136263
landed. The recommit fixed those.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D139509
The Zfhmin subset only has FLH, FSH, FMV.X.H, FMV.H.X, FCVT.S.H, and FCVT.H.S.
If the D extension is present, the FCVT.D.H and FCVT.H.D instructions are also included.
Since most instructions are not included for Zfhmin, so most operations are promoted.
The patch primarily about making f16 a legal type.
RISC-V ISA info:
https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D139391
The patch made VectorLegalizer expand ISD::VP_FSHL and ISD::VP_FSHR to
achieve the codegen.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D138379
This patch implements shouldFoldSelectWithIdentityConstant for RISCV. It would try to generate vmerge after the binary instruction and let them folded to maksed instruction later.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D131551
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
At the IR level, we generally assume that constants are free to materialize. However, for RISCV due to some quirks of the ISA, materializing arbitrary constants can be rather expensive. We frequently fallback to constant pool loads.
We've been slowly moving in the direction of modeling the cost of the remat as part of the instruction cost. This has the effect of disincentivizing vectorization - mostly SLP - when we'd have to materialize an expensive constant.
We need better modeling of which constants are expensive and not, but the moment let's be consistent with how we model arithmetic and memory instructions. The difference between the two is that arithmetic can sometimes fold a splat operation which stores can not.
Differential Revision: https://reviews.llvm.org/D138941
This patch reduces the number of unpredictable branches
(select (x < 0), y, z) -> x >> (XLEN - 1) & (y - z) + z
(select (x >= 0), y, z) -> x >> (XLEN - 1) & (z - y) + y
Reviewed By: craig.topper, reames
Differential Revision: https://reviews.llvm.org/D137949
So we have the opportunity to fold splat into .vx instruction as what
D101138 has done. If failed, we can select zero-stride vector load
again.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D138101
This patch add the support of RISCV Zca ext
`Zca` is a subset of C extension instructions that are compatible with the Zc extension.
So this patch implements Zca code generation with reference to the C extension and sets the 2-byte alignment for the Zca extension, just like C extension does.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D130483
Current lowerVECTOR_SHUFFLEAsVSlidedown only seeks whether input are
EXTRACT_SUBVECTOR and their source are same. The commit will make the
function seek input and their source until they are not
EXTRACT_SUBVECTOR.
Differential Revision: https://reviews.llvm.org/D138025
A target can return if a misaligned access is 'fast' as defined
by the target or not. In reality there can be different levels
of 'fast' and 'slow'. This patch changes the boolean 'Fast'
argument of the allowsMisalignedMemoryAccesses family of functions
to an unsigned representing its speed.
A target can still define it as it wants and the direct translation
of the current code uses 0 and 1 for current false and true. This
makes the change an NFC.
Subsequent patch will start using an actual value of speed in
the load/store vectorizer to compare if a vectorized access going
to be not just fast, but not slower than before.
Differential Revision: https://reviews.llvm.org/D124217
Type legalization will want to turn (srl X, Y) into RISCVISD::SRLW,
which will prevent us from using a BEXT instruction.
This is similar to what we do for (i32 (and (srl X, Y), 1)).
nearbyint has the property to execute without exception.
For not modifying fflags, the patch added new machine opcode
PseudoVFROUND_NOEXCEPT_V that expands vfcvt.x.f.v and vfcvt.f.x.v between a pair
of frflags and fsflags.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D137685
The patch also added function expandVPBSWAP to expand ISD::VP_BSWAP nodes.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D137928
We may form a zero-stride vector load when lowering gather to strided
load. As what D137699 has done, we use `load+splat` for this form if
there is no optimized implementation.
We restrict this to unmasked loads currently in consideration of the
complexity of hanlding all falses masks.
Reviewed By: reames
Differential Revision: https://reviews.llvm.org/D137931
Current getDefaultVLOps can only deduce VL from a MVT. However,
sometimes users have already known VL value. This commit will provide a
uniform interface to get VL instead of calling DAG.getConstant.
Differential Revision: https://reviews.llvm.org/D138003
This adds a RISCVISD::ABSW to remember that we started with an i32
abs. Previously we used a DAG combine of (sext_inreg (abs)) to
delay emitting a freeze from type legalization in order to make
ComputeNumSignBits optimizations work on other promoted nodes.
This new approach always uses negw+max even if the result doesn't
need to be sign extended. This helps the RISCVSExtWRemoval pass
if the sext.w is in another basic block.
All ISD::BSWAP nodes are not customized lowered in RISC-V now, so the patch
removed dead code for ISD::BSWAP in LowerOperation.
Reviewed By: craig.topper
Differential Revision: https://reviews.llvm.org/D137907
