Sam Tebbs 795e35a653 Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744) ...

This relands the reverted #120721 with a fix for cases where neither
reduction operand are the reduction phi. Only
63114239cc8d26225a0ef9920baacfc7cc00fc58 and
63114239cc8d26225a0ef9920baacfc7cc00fc58 are new on top of the reverted
PR.

---------

Co-authored-by: Nicholas Guy <nicholas.guy@arm.com>

2025-01-13 11:20:35 +00:00

..

aarch64-predication.ll

[LoopVectorize][NFC] Rewrite tests to check output of vplan cost model (#113697)

2024-11-19 08:55:39 +00:00

aarch64-unroll.ll

…

arbitrary-induction-step.ll

…

arith-fp-frem-costs.ll

[LoopVectorize][NFC] Fix arith-fp-frem-costs.ll test to use new vplan cost model (#120742)

2025-01-06 10:26:51 +00:00

arm64-unroll.ll

…

backedge-overflow.ll

…

blend-costs.ll

[VPlan] Handle some VPInstructions in may{Read,Write}FromMemory (#120058)

2025-01-08 15:17:26 +08:00

call-costs.ll

[VPlan] Remove loop region in optimizeForVFAndUF. (#108378)

2025-01-05 15:50:42 +00:00

clamped-trip-count.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

conditional-branches-cost.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

cost-no-valid-vplans-built.ll

…

deterministic-type-shrinkage.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

divs-with-scalable-vfs.ll

Recommit "[VPlan] Try to narrow wide and replicating recipes to uniform recipes."

2025-01-12 20:10:28 +00:00

eliminate-tail-predication.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

epilog-vectorization-factors.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

epilog-vectorization-widen-inductions.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

extend-vectorization-factor-for-unprofitable-memops.ll

…

extractvalue-no-scalarization-required.ll

[LV] Update checks to actually check VPlan for correct function.

2024-12-07 12:05:17 +00:00

first-order-recurrence-fold-tail.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

first-order-recurrence.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

fixed-order-recurrence.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

force-target-instruction-cost.ll

[VPlan] Handle some VPInstructions in may{Read,Write}FromMemory (#120058)

2025-01-08 15:17:26 +08:00

fully-unrolled-cost.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

gather-cost.ll

…

gather-do-not-vectorize-addressing.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

i1-reg-usage.ll

…

induction-costs-sve.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

induction-costs.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

induction-trunc.ll

…

interleave_count_for_estimated_tc.ll

…

interleave_count_for_known_tc.ll

…

interleave-allocsize-not-equal-typesize.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

interleaved_cost.ll

[LoopVectorize] Make needsExtract notice scalarized instructions (#119720)

2025-01-02 14:31:36 +00:00

interleaved-store-of-first-order-recurrence.ll

[VPlan] Don't created GEP x, 0 for interleave group pointers.

2024-10-08 12:08:13 +01:00

interleaved-vs-scalar.ll

…

interleaving-load-store.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

interleaving-reduction.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

intrinsiccost.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

invalid-costs.ll

[VPlan] Return cost of PHI for scalar VFs in computeCost for FORs.

2024-11-21 21:11:21 +00:00

invariant-replicate-region.ll

…

lit.local.cfg

…

loop-vectorization-factors.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

loopvectorize_pr33804_double.ll

…

low_trip_count_predicates.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

low_trip_memcheck_cost.ll

…

masked-call.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

masked-op-cost.ll

[LoopVectorize][NFC] Rewrite tests to check output of vplan cost model (#113697)

2024-11-19 08:55:39 +00:00

max-vf-for-interleaved.ll

…

maximize-bandwidth-invalidate.ll

[LoopVectorize][NFC] Rewrite tests to check output of vplan cost model (#113697)

2024-11-19 08:55:39 +00:00

mul-simplification.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

neon-gather-cost.ll

…

neoverse-epilogue-vect.ll

[AArch64][LV] Set MaxInterleaving to 4 for Neoverse V2 and V3 (#100385)

2024-11-20 09:33:39 +00:00

no_vector_instructions.ll

[LoopVectorize][NFC] Rewrite tests to check output of vplan cost model (#113697)

2024-11-19 08:55:39 +00:00

nontemporal-load-store.ll

…

outer_loop_prefer_scalable.ll

[VPlan] Omit zero add in VPWidenIntOrFpInductionRecipe (#119668)

2024-12-16 11:55:48 +08:00

outer_loop_test1_no_explicit_vect_width.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

Oz-and-forced-vectorize.ll

…

partial-reduce-dot-product-epilogue.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

partial-reduce-dot-product-mixed.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

partial-reduce-dot-product-neon.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

partial-reduce-dot-product.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

partial-reduce-no-dotprod.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

pr31900.ll

…

pr33053.ll

…

pr36032.ll

…

pr46950-load-cast-context-crash.ll

…

pr56627.ll

…

pr60831-sve-inv-store-crash.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

pr73894.ll

…

predication_costs.ll

…

prefer-fixed-if-equal-to-scalable.ll

…

reduction-recurrence-costs-sve.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

reduction-small-size.ll

…

reg-usage.ll

…

runtime-check-size-based-threshold.ll

…

runtime-check-trip-count-decisions.ll

…

scalable-alloca.ll

…

scalable-avoid-scalarization.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

scalable-call.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

scalable-fp-ext-trunc-illegal-type.ll

[VPlan] Propagate all GEP flags (#119899)

2024-12-17 13:48:50 +01:00

scalable-predicate-instruction.ll

…

scalable-reduction-inloop-cond.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

scalable-reductions-tf.ll

…

scalable-reductions.ll

…

scalable-strict-fadd.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

scalable-struct-return.ll

[LV] Teach LoopVectorizationLegality about struct vector calls (#119221)

2025-01-09 09:27:29 +00:00

scalable-vectorization-cost-tuning.ll

[AArch64] Change the default vscale-for-tuning to 1. (#117174)

2024-11-29 17:41:05 +00:00

scalable-vectorization.ll

Revert "[AArch64][SVE] Enable max vector bandwidth for SVE" (#112873)

2024-10-18 11:05:55 +01:00

scalable-vf-hint.ll

[AArch64] Change the default vscale-for-tuning to 1. (#117174)

2024-11-29 17:41:05 +00:00

scalar_interleave.ll

…

scalarize-store-with-predication.ll

…

sdiv-pow2.ll

…

select-costs.ll

Recommit "[LV][VPlan] Remove any-of reduction from precomputeCost. NFC (#117109)" (#117289)

2024-11-28 15:07:36 +08:00

simple_early_exit.ll

[LV] Fixup check lines after 13107cb094.

2024-12-18 09:37:30 +00:00

smallest-and-widest-types.ll

…

sme-vectorize.ll

…

store-costs-sve.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

streaming-compatible-sve-no-maximize-bandwidth.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

streaming-vectorization.ll

…

strict-fadd-cost.ll

…

strict-fadd-vf1.ll

…

strict-fadd.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

sve2-histcnt-epilogue.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve2-histcnt-no-scalar-interleave.ll

[InstCombine] Infer nusw + nneg -> nuw for getelementptr (#111144)

2024-12-05 14:36:40 +01:00

sve2-histcnt-outerloop-scevaddrec.ll

…

sve2-histcnt-too-many-deps.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve2-histcnt-vplan.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve2-histcnt.ll

Revert "[InstCombine] Infer nuw for gep inbounds from base of object" (#120460)

2024-12-18 19:06:34 +01:00

sve-basic-vec.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-cond-inv-loads.ll

[InstCombine] Infer nusw + nneg -> nuw for getelementptr (#111144)

2024-12-05 14:36:40 +01:00

sve-epilog-vect-inloop-reductions.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-epilog-vect-reductions.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-epilog-vect-strict-reductions.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-epilog-vect-vscale-tune.ll

[AArch64][LV] Set MaxInterleaving to 4 for Neoverse V2 and V3 (#100385)

2024-11-20 09:33:39 +00:00

sve-epilog-vect.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-extract-last-veclane.ll

…

sve-fneg.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-gather-scatter-cost.ll

[LAA] Don't require Stride == 1/-1 for inbounds pointer AddRecs nowrap. (#113126)

2024-11-05 22:45:56 +01:00

sve-gather-scatter.ll

[InstCombine] Infer nusw + nneg -> nuw for getelementptr (#111144)

2024-12-05 14:36:40 +01:00

sve-illegal-type.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

sve-inductions-unusual-types.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-inductions.ll

…

sve-interleaved-accesses.ll

Revert "[LV]: Teach LV to recursively (de)interleave. (#89018)"

2024-12-31 03:12:24 +05:00

sve-interleaved-masked-accesses.ll

Revert "[LV]: Teach LV to recursively (de)interleave. (#89018)"

2024-12-31 03:12:24 +05:00

sve-inv-loads.ll

…

sve-inv-store.ll

…

sve-invalid-costs.ll

…

sve-large-strides.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-live-out-pointer-induction.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-low-trip-count.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-masked-loadstore.ll

…

sve-multiexit.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-runtime-check-size-based-threshold.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-scalable-load-in-loop.ll

…

sve-select-cmp.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-strict-fadd-cost.ll

[AArch64] Change the default vscale-for-tuning to 1. (#117174)

2024-11-29 17:41:05 +00:00

sve-tail-folding-cost.ll

…

sve-tail-folding-forced.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-tail-folding-option.ll

…

sve-tail-folding-optsize.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-tail-folding-overflow-checks.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-tail-folding-reductions.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-tail-folding-unroll.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

sve-tail-folding.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-type-conv.ll

…

sve-vector-reverse-mask4.ll

…

sve-vector-reverse.ll

[VPlan] Remove reverse() of predecessors from VPInstruction::generate.

2024-12-17 20:44:32 +00:00

sve-vfabi.ll

…

sve-widen-extractvalue.ll

…

sve-widen-gep.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

sve-widen-phi.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

synthesize-mask-for-call.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

tail-fold-uniform-memops.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

tail-folding-styles.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00

type-shrinkage-insertelt.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

type-shrinkage-zext-costs.ll

[VPlan] Remove stray space when printing VPWidenCastRecipe.

2024-12-24 20:23:48 +00:00

uniform-args-call-variants.ll

[InstCombine] Infer nusw + nneg -> nuw for getelementptr (#111144)

2024-12-05 14:36:40 +01:00

unsafe-vf-hint-remark.ll

…

veclib-calls-libsystem-darwin.ll

[aarch64] atan2 intrinsic lowering (p5) (#112611)

2024-10-24 17:53:12 -07:00

veclib-function-calls.ll

[AArch64] Change the default vscale-for-tuning to 1. (#117174)

2024-11-29 17:41:05 +00:00

veclib-intrinsic-calls.ll

[aarch64] atan2 intrinsic lowering (p5) (#112611)

2024-10-24 17:53:12 -07:00

vector-call-linear-args.ll

…

vector-reverse-mask4.ll

[LLVM][IR] Use splat syntax when printing Constant[Data]Vector. (#112548)

2024-11-06 11:53:33 +00:00

vector-reverse.ll

…

vplan-printing.ll

Reland "[LoopVectorizer] Add support for partial reductions" with non-phi operand fix. (#121744)

2025-01-13 11:20:35 +00:00

widen-call-with-intrinsic-or-libfunc.ll

[VPlan] Compute induction end values in VPlan. (#112145)

2024-12-29 19:05:08 +00:00

wider-VF-for-callinst.ll

[LLVM][IR] Use splat syntax when printing ConstantExpr based splats. (#116856)

2024-11-21 11:21:12 +00:00