We want to use profile inference (**profi**) in BOLT for stale profile matching.
To this end, I am making a few changes modifying the interface of the algorithm.
This is the first change for existing usages of profi (e.g., CSSPGO):
- introducing an object holding the algorithmic parameters;
- some renaming of existing options;
- dropped unused option, SampleProfileInferEntryCount, as we don't plan to change its default value;
- no changes in the output / tests.
Reviewed By: hoy
Differential Revision: https://reviews.llvm.org/D134756
Preserve the alignment of the original atomicrmw, rather than using
the ABI alignment.
The same problem exists for loads, but that code is being removed
in D141277 anyway.
This is a patch to fix duplicated dbg.values in the JumpThreading pass not
pointing towards their local value, and instead towards the variable in the
original block.
JumpThreadingPass::cloneInstructions is the changed function to target metadata
as well as normal cloned values.
Reviewed By: jmorse, StephenTozer
Differential Revision: https://reviews.llvm.org/D140006
While demanded bits constant shrinking appears to prevent this in
practice right now, it is principally possible for C2 to have
set bits that are known not-needed (zeroable). See: D140858
`+` will overflow here, `|` will get the right logic.
Differential Revision: https://reviews.llvm.org/D141089
Adds a whitespace in a debug message before printing out a
value in the SSAUpdaterBulk.
Without this, debugging can end up a bit cumbersome.
Differential Revision: https://reviews.llvm.org/D141262
Jump threading can replace select and unconditional branch with
conditional branch, but when doing so loses profile information.
This destructive transform can eventually lead to a performance
degradation due to folding of branches in
shouldFoldCondBranchesToCommonDestination as branch probabilities
are no longer known.
Patch by Roman Paukner!
Differential Revision: https://reviews.llvm.org/D138132
Reviewed By: mkazantsev
This patch introduces a new AA `AAUnderlyingObjects`. It is basically like a wrapper
AA of the function `AA::getAssumedUnderlyingObjects`, but it can recursively do
query if the underlying object is an indirect access, such as a phi node or a select
instruction.
Reviewed By: jdoerfert
Differential Revision: https://reviews.llvm.org/D141164
Follow-up to:
6c39a3aae1
That converted a pattern with ashr directly to icmp+zext, and
this updates the pattern that we used to convert to.
This canonicalizes to icmp for better analysis in the minimum case
and shortens patterns where the source type is not the same as dest type:
https://alive2.llvm.org/ce/z/tpXJ64https://alive2.llvm.org/ce/z/dQ405O
This requires an adjustment to an icmp transform to avoid infinite looping.
The new mask represents the order, not the mask itself. At first, need
to treat as the order, convert to mask and only after that reorder
gathered scalars to build correct clustered order.
Differential Revision: https://reviews.llvm.org/D141161
Prior to this patch, variadic DIExpressions (i.e. ones that contain
DW_OP_LLVM_arg) could only be created by salvaging debug values to create
stack value expressions, resulting in a DBG_VALUE_LIST being created. As of
the previous patch in this patch stack, DBG_INSTR_REF's syntax has been
changed to match DBG_VALUE_LIST in preparation for supporting variadic
expressions. This patch adds some minor changes needed to allow variadic
expressions that aren't stack values to exist, and allows variadic expressions
that are trivially reduceable to non-variadic expressions to be handled
similarly to non-variadic expressions.
Reviewed by: jmorse
Differential Revision: https://reviews.llvm.org/D133926
Non-throwing inline asm infers the nounwind attribute in
instcombine. Thus, it can be handled in the same manner as
non-throwing target functions are generally. Further special casing is
unnecessary complexity.
Need to include the cost of the initial insertelement to the cost of the
broadcasts. Also, need to adjust the cost of the gather/buildvector if
the element is inserted into poison/undef vector.
Differential Revision: https://reviews.llvm.org/D140498
This is a recurring pattern: We want to find the nearest common
dominator (instruction) for two instructions, but currently only
provide an API for the nearest common dominator of two basic blocks.
Add an overload that accepts and return instructions.
Similar to D138404, we were not guarding against extra uses of the Mul.
In most cases other checks would catch the issue due to unsupported
instructions in the outer loop, but certain non-canonical loop forms
could still get through.
Fixes#59339
Differential Revision: https://reviews.llvm.org/D141114
When fetching allocation sizes, we almost always want to have the
size in bytes, but we were only providing an InBits API. Also add
the corresponding byte-based conjugate to save some *8 and /8
juggling everywhere.
This patch adds metadata to disable runtime unrolling to the vectorized
loop. If runtime unrolling/interleaving is considered profitable, LV
will interleave the loop directly. There should be no need to perform
runtime unrolling at a later stage.
Note that we already add metadata to disable runtime unrolling to the
scalar loop after vectorization.
The additional unrolling unnecessarily increases code size and compile
time. In addition to that we have several bug reports of unncessary
runtime unrolling for vectorized loops, e.g. PR40961
Compile-time improvements:
NewPM-O3: -1.04%
NewPM-ReleaseThinLTO: -0.59%
NewPM-ReleaseLTO-g: -0.97%
https://llvm-compile-time-tracker.com/compare.php?from=ce1be13a868d0f8afa367975558c1a6175cce33a&to=78bc2e67f22e9e10e61cdb6cdac4bb857d95eb1b&stat=instructions:uFixes#40306.
Reviewed By: lebedev.ri, nikic
Differential Revision: https://reviews.llvm.org/D115261
NFC-ish. There is a functional change but the outputs are semantically
identical. Where we might've before replaced one operand with undef (which
means "this is a kill location marker") the use of `setKillLocation` will
replace all location operands with `undef` (which also means "this is a kill
location marker").
Related to https://discourse.llvm.org/t/auto-undef-debug-uses-of-a-deleted-value
Reviewed By: StephenTozer
Differential Revision: https://reviews.llvm.org/D140904
Make a separate routine for GEPs cost calculation and make
the approach uniform across load, store and GEP tree nodes.
Additional issue fixed is GEP cost savings were applied twice
for ScatterVectorize nodes (aka gather load) making them look
unrealistically profitable for vectorization.
Differential Revision: https://reviews.llvm.org/D140789
Use deduction guides instead of helper functions.
The only non-automatic changes have been:
1. ArrayRef(some_uint8_pointer, 0) needs to be changed into ArrayRef(some_uint8_pointer, (size_t)0) to avoid an ambiguous call with ArrayRef((uint8_t*), (uint8_t*))
2. CVSymbol sym(makeArrayRef(symStorage)); needed to be rewritten as CVSymbol sym{ArrayRef(symStorage)}; otherwise the compiler is confused and thinks we have a (bad) function prototype. There was a few similar situation across the codebase.
3. ADL doesn't seem to work the same for deduction-guides and functions, so at some point the llvm namespace must be explicitly stated.
4. The "reference mode" of makeArrayRef(ArrayRef<T> &) that acts as no-op is not supported (a constructor cannot achieve that).
Per reviewers' comment, some useless makeArrayRef have been removed in the process.
This is a follow-up to https://reviews.llvm.org/D140896 that introduced
the deduction guides.
Differential Revision: https://reviews.llvm.org/D140955
The validation of vplans could fail if an inloop reduction was created
with a block-in mask that did not dominate the reduction. This makes
sure that the insert point is set when creating the mask, to ensure it
dominates the reduction.
Differential Revision: https://reviews.llvm.org/D141003
Assuming that collecting frame allocas doesn't modify CFG we can safely
move DominatorTree construction outside loop and avoid expensive computations.
Differential Revision: https://reviews.llvm.org/D140818
Fixes https://github.com/llvm/llvm-project/issues/58565
The previous implementation visits operands in pre-order, but this does
not guarantee an instruction is visited before its uses. This can cause
instructions to be copied in the incorrect order. For example:
```
a = ...
b = add a, 1
c = add a, b
d = add b, a
```
Pre-order visits does not guarantee the order in which `a` and `b` are
visited. LoopUnrollAndJam may incorrectly insert `b` before `a`.
This patch implements post-order visits. By visiting dependencies first,
we guarantee that an instruction's dependencies are visited first.
Differential Revision: https://reviews.llvm.org/D140255
Switch simplification could sometimes fail to notice when an
intermediate case removal caused the switch condition to become
constant. This would cause the switch to be simplified into a
conditional branch rather than a direct branch.
Most of the time this didn't matter, except that occasionally
downstream parts of SimplifyCFG expect tautological branches to
already have been eliminated. The missed handling in switch
simplification would cause an assertion failure in the downstream
code.
Triggering the assertion failure is fairly sensitive to the exact
order of various simplifications.
Fixes https://github.com/llvm/llvm-project/issues/59768
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D140831
This reverts commit aa2414729e.
Previously-valid IR from a tensorflow test case (as shown on the
Diffusion revision for aa2414729e) started
hanging in the loop-vectorize pass. Reverting to keep everyone working.
analysis.
Missed the analysis of the shuffle mask when trying to analyze the
operands of the shuffle instruction during peeking through shuffle
instructions.
We shouldn't penalize instructions that have extra flags.
Drop the poison-generating flags if needed instead of bailing out.
This makes canonicalization/optimization more uniform.
There is a chance that dropping flags will cause some
other transform to not fire, but we added a preliminary
patch to avoid that with:
f0faea5714
See D140665 for more details.
