Add `requiresReplacedValues` and `visitReplacedValues` methods to
`PromotableOpInterface`. These methods allow `PromotableOpInterface` ops
to transforms definitions mutated by a `store`.
This change is necessary to correctly handle the promotion of
`LLVM_DbgDeclareOp`.
---------
Co-authored-by: Théo Degioanni <30992420+Moxinilian@users.noreply.github.com>
Composite pass allows to run sequence of passes in the loop until fixed
point or maximum number of iterations is reached. The usual candidates
are canonicalize+CSE as canonicalize can open more opportunities for CSE
and vice-versa.
There is no good way to report detailed errors from inside
`Pass::initializeOptions` function as context may not be available at
this point and writing directly to `llvm::errs()` is not composable.
See
https://github.com/llvm/llvm-project/pull/87166#discussion_r1546426763
* Add error handler callback to `Pass::initializeOptions`
* Update `PassOptions::parseFromString` to support custom error stream
instead of using `llvm::errs()` directly.
* Update default `Pass::initializeOptions` implementation to propagate
error string from `parseFromString` to new error handler.
* Update `MapMemRefStorageClassPass` to report error details using new
API.
Before deleting the block we need to drop uses to the surrounding args.
If this is not performed dialect conversion failures can result in a
failure to remove args (despite the block having no remaining uses).
I believe the existing check to determine if an operand should be added
is incorrect: `operand.use_empty() || operand.hasOneUse()`. This is
because these checks do not take into account the fact that the op is
being deleted. It hasn't been deleted yet, so `operand.use_empty()`
cannot be true, and `operand.hasOneUse()` may be true if the op being
deleted is the only user of the operand and it only uses it once, but it
will fail if the operand is used more than once (e.g. something like
`add %0, %0`).
Instead, check if the op being deleted is the only _user_ of the
operand. If so, add the operand to the worklist.
Fixes#86765
Computing the inlinling profitability can be costly due to walking the
graph when counting the number of operations.
This PR addresses that by returning early if the threshold is set to
never or always inline.
This commit expends the Mem2Reg and SROA interface methods with passed
in handles to a `DataLayout` structure. This is done to avoid
superfluous retreiving of data layouts during each conversion of
intrinsics.
This change, additionally, enables subsequent changes to make the LLVM
dialect implementation of these interfaces type agnostic.
This commit changes MLIR's SROA implementation back from being pattern
based into a full pass. This is beneficial for upcoming changes that
rely more heavily on the datalayout.
Unfortunately, this change required substantial test changes, as the
IRBuilder no cleans up the IR.
This commit changes MLIR's Mem2Reg implementation back from being
pattern based into a full pass. Using Mem2Reg as a pattern is
wasteful, as each application can invalidate the dominance info.
Applying changes in bulk allows for reuse of the same dominance info.
Unfortunately, this requires some test changes, due to the `IRBuilder`
not simplifying IR.
Discussion at https://discourse.llvm.org/t/inliner-cost-model/2992
This change adds a callback that reports whether inlining
of the particular call site (communicated via ResolvedCall argument)
is profitable or not. The default MLIR inliner pass behavior
is unchanged, i.e. the callback always returns true.
This callback may be used to customize the inliner behavior
based on the target specifics (like target instructions costs),
profitability of the inlining for further optimizations
(e.g. if inlining may enable loop optimizations or scalar optimizations
due to object shape propagation), optimization levels (e.g. -Os inlining
may be quite different from -Ofast inlining), etc.
One of the questions is whether the ResolvedCall entity represents
enough of the context for the custom inlining models to come up with
the profitability decision. I think we can start with this and
extend it as necessary.
---------
Co-authored-by: Mehdi Amini <joker.eph@gmail.com>
#82629 added additional overloads to `replaceAllUsesWith` and
`replaceUsesWithIf`. This caused a build breakage with MSVC when called
with ops that can implicitly convert to `Value`.
```
external/llvm-project/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp(881): error C2666: 'mlir::RewriterBase::replaceAllUsesWith': 2 overloads have similar conversions
external/llvm-project/mlir/include\mlir/IR/PatternMatch.h(631): note: could be 'void mlir::RewriterBase::replaceAllUsesWith(mlir::Operation *,mlir::ValueRange)'
external/llvm-project/mlir/include\mlir/IR/PatternMatch.h(626): note: or 'void mlir::RewriterBase::replaceAllUsesWith(mlir::ValueRange,mlir::ValueRange)'
external/llvm-project/mlir/include\mlir/IR/PatternMatch.h(616): note: or 'void mlir::RewriterBase::replaceAllUsesWith(mlir::Value,mlir::Value)'
external/llvm-project/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp(882): note: while trying to match the argument list '(mlir::tensor::ExtractSliceOp, T)'
with
[
T=mlir::Value
]
```
Note: The LLVM build bots (Linux and Windows) did not break, this seems
to be an issue with `Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe`.
This change renames the newly added overloads to `replaceAllOpUsesWith`
and `replaceOpUsesWithIf`.
This commit adds two new notifications to `RewriterBase::Listener`:
* `notifyPatternBegin`: Called when a pattern application begins during
a greedy pattern rewrite or dialect conversion.
* `notifyPatternEnd`: Called when a pattern application finishes during
a greedy pattern rewrite or dialect conversion.
The listener infrastructure already provides a `notifyMatchFailure`
callback that notifies about the reason for a pattern match failure. The
two new notifications provide additional information about pattern
applications.
This change is in preparation of improving the handle update mechanism
in the `apply_conversion_patterns` transform op.
This commit adds listener support to the dialect conversion. Similarly
to the greedy pattern rewrite driver, an optional listener can be
specified in the configuration object.
Listeners are notified only if the dialect conversion succeeds. In case
of a failure, where some IR changes are first performed and then rolled
back, no notifications are sent.
Due to the fact that some kinds of rewrite are reflected in the IR
immediately and some in a delayed fashion, there are certain limitations
when attaching a listener; these are documented in `ConversionConfig`.
To summarize, users are always notified about all rewrites that
happened, but the notifications are sent all at once at the very end,
and not interleaved with the actual IR changes.
This change is in preparation improvements to
`transform.apply_conversion_patterns`, which currently invalidates all
handles. In the future, it can use a listener to update handles
accordingly, similar to `transform.apply_patterns`.
During block signature conversion, a new block is inserted and ops are
moved from the old block to the new block. This commit changes the
implementation such that ops are moved in bulk (`splice`) instead of
one-by-one; that's what `splitBlock` is doing.
This also makes it possible to pass the new block argument types
directly to `createBlock` instead of using `addArgument` (which bypasses
the rewriter). This doesn't change anything from a technical point of
view (there is no rewriter API for adding arguments at the moment), but
the implementation reads a bit nicer.
* `replaceOp` replaces all uses of the original op and erases the old
op.
* `replaceAllUsesWith` replaces all uses of the original op/value/block.
It does not erase any IR.
This commit renames `replaceOpWithIf` to `replaceUsesWithIf`.
`replaceOpWithIf` was a misnomer because the function never erases the
original op. Similarly, `replaceOpWithinBlock` is renamed to
`replaceUsesWithinBlock`. (No "operation replaced" is sent because the
op is not erased.)
Also improve comments.
This is just code refactoring done as a preparation for adding
MLIR inliner cost model hook(s).
Related discussion: https://discourse.llvm.org/t/inliner-cost-model/2992
The logic of SCC-based MLIR inliner is separated into the Inliner
implementation. The MLIR inliner pass becomes, well, just a pass
that invokes the SCC-based MLIR inliner.
When compiling for GCC 8.x (< 8.4), SFINAE is disabled for
iterator_range constructor causing ambiguous resolution to construct an
OperandRange from a MutableOperatorRange, even in the presence of a
static_cast<OperatorRange>. This adds an explicit conversion method to
lift the ambiguity.
Tested with a full MLIR build with GCC 8.3.
notifyBlockInserted can be called when inserting a block in a region before
the op is built (like when building a scf::ForOp). This make us defensive
by checking the parent op before printing it.
The dialect conversion uses a `SingleEraseRewriter` to ensure that an
op/block is not erased twice. This can happen during the "commit" phase
when an unresolved materialization is inserted into a block and the
enclosing op is erased by the user. In that case, the unresolved
materialization should not be erased a second time later in the "commit"
phase.
This problem cannot happen during "rollback", so ops/block can be erased
directly without using the rewriter. With this change, the
`SingleEraseRewriter` is used only during "commit"/"cleanup". At that
point, the dialect conversion is guaranteed to succeed and no rollback
can happen. Therefore, it is not necessary to store the number of erased
IR objects (because we will never "reset" the rewriter to previous a
previous state).
When a block signature is converted during dialect conversion, a
`BlockTypeConversionRewrite` object is stored in the stack of rewrites.
Such an object represents multiple steps:
- Splitting the old block, i.e., creating a new block and moving all
operations over.
- Rewriting block arguments.
- Erasing the old block.
We have dedicated `IRRewrite` objects that represent "creating a block",
"moving an op" and "erasing a block". This commit reuses those rewrite
objects, so that there is less work to do in
`BlockTypeConversionRewrite::rollback` and
`BlockTypeConversionRewrite::commit`/`cleanup`.
Note: This change is in preparation of adding listener support to the
dialect conversion. The less work is done in a `commit` function, the
fewer notifications will have to be sent.
This commit adds a new `ConversionConfig` struct that allows users to
customize the dialect conversion. This configuration is similar to
`GreedyRewriteConfig` for the greedy pattern rewrite driver.
A few existing options are moved to this objects, simplifying the
dialect conversion API.
This is a re-upload of #82250. The Windows build breakage was fixed in #83768.
This reverts commit 60fbd60501.
This commit fixes a bug in a dialect conversion. Currently, when a block
is replaced via a signature conversion, the block is erased during the
"commit" phase. This is problematic because the block arguments may
still be referenced internal data structures of the dialect conversion
(`mapping`). Blocks should be treated same as ops: they should be erased
during the "cleanup" phase.
Note: The test case fails without this fix when running with ASAN, but
may pass when running without ASAN.
When splitting a block during a dialect conversion, a
`SplitBlockRewrite` object is stored in the dialect conversion state.
This commit removes `SplitBlockRewrite`. Instead, a combination of
`CreateBlockRewrite` and multiple `MoveOperationRewrite` is used.
This change simplifies the internal state of the dialect conversion and
is also needed to properly support listeners.
`RewriteBase::splitBlock` is now no longer virtual. All necessary
information for committing/rolling back a split block rewrite can be
deduced from `Listener::notifyBlockInserted` and
`Listener::notifyOperationInserted` (which is also called when moving an
operation).
The dialect conversion maintains sets of "ignored" and "replaced" ops.
This change simplifies the two sets, such that all nested ops are
included. (This was previously not the case and sometimes only the
parent op was included.)
This change allows for more aggressive assertions to prevent incorrect
rewriter API usage. E.g., accessing ops/blocks/regions within an erased
op.
A concrete example: I have seen conversion patterns in downstream
projects where an op is replaced with a new op, and the region of the
old op is afterwards inlined into the newly created op. This is invalid
rewriter API usage: ops that were replaced/erased should not be
accessed. Nested ops will be considered "ignored", even if they are
moved to a different region after the region's parent op was erased
(which is illegal API usage). Instead, create a new op, inline the
regions, then replace the old op with the new op.
#83023 fixed a performance regression related to "ignored" ops. This
broke some downstream projects that access ops after they were replaced
(an API violation). This change restores the original behavior before
#83023 (but without the performance regression), to give downstream
users more time to fix their code.
The dialect conversion does not directly erase ops that are
replaced/erased with a rewriter. Instead, the op stays in place and is
erased at the end if the dialect conversion succeeds. However, ops that
were replaced/erased are ignored from that point on.
#81757 introduced a compile time regression that made the check whether
an op is ignored or not more expensive. Whether an op is ignored or not
is queried many times throughout a dialect conversion, so the check must
be fast.
After this change, replaced ops are stored in the `ignoredOps` set. This
also simplifies the dialect conversion a bit.
This is a follow-up to #82333. It is possible that the target block of a
`BlockTypeConversionRewrite` is detached, so the `MLIRContext` cannot be
taken from the block.
`ConversionPatternRewriterImpl` no longer maintains a reference to the
respective `ConversionPatternRewriter`. An `MLIRContext` is sufficient.
This commit simplifies the internal state of
`ConversionPatternRewriterImpl`.
This commit adds a new `ConversionConfig` struct that allows users to
customize the dialect conversion. This configuration is similar to
`GreedyRewriteConfig` for the greedy pattern rewrite driver.
A few existing options are moved to this objects, simplifying the
dialect conversion API.
`ConversionPatternRewriter` objects should not be constructed outside of
dialect conversions. Some IR modifications performed through a
`ConversionPatternRewriter` are reflected in the IR in a delayed fashion
(e.g., only when the dialect conversion is guaranteed to succeed). Using
a `ConversionPatternRewriter` outside of the dialect conversion is
incorrect API usage and can bring the IR in an inconsistent state.
Migration guide: Use `IRRewriter` instead of
`ConversionPatternRewriter`.
This commit is a refactoring of the dialect conversion. The dialect
conversion maintains a list of "IR rewrites" that can be committed (upon
success) or rolled back (upon failure).
This commit turns the creation of unresolved materializations
(`unrealized_conversion_cast`) into `IRRewrite` objects. After this
commit, all steps in `applyRewrites` and `discardRewrites` are calls to
`IRRewrite::commit` and `IRRewrite::rollback`.
This commit is a refactoring of the dialect conversion. The dialect
conversion maintains a list of "IR rewrites" that can be committed (upon
success) or rolled back (upon failure).
Until now, the dialect conversion kept track of "op creation" in
separate internal data structures. This commit turns "op creation" into
an `IRRewrite` that can be committed and rolled back just like any other
rewrite. This commit simplifies the internal state of the dialect
conversion.
This commit is a refactoring of the dialect conversion. The dialect
conversion maintains a list of "IR rewrites" that can be committed (upon
success) or rolled back (upon failure).
Until now, op replacements and block argument replacements were kept
track in separate data structures inside the dialect conversion. This
commit turns them into `IRRewrite`s, so that they can be committed or
rolled back just like any other rewrite. This simplifies the internal
state of the dialect conversion.
Overview of changes:
* Add two new rewrite classes: `ReplaceBlockArgRewrite` and
`ReplaceOperationRewrite`. Remove the `OpReplacement` helper class; it
is now part of `ReplaceOperationRewrite`.
* Simplify `RewriterState`: `numReplacements` and `numArgReplacements`
are no longer needed. (Now being kept track of by `numRewrites`.)
* Add `IRRewrite::cleanup`. Operations should not be erased in `commit`
because they may still be referenced in other internal state of the
dialect conversion (`mapping`). Detaching operations is fine.
* `trackedOps` are now updated during the "commit" phase instead of
after applying all rewrites.
This commit is a refactoring of the dialect conversion. The dialect
conversion maintains a list of "IR rewrites" that can be committed (upon
success) or rolled back (upon failure).
Until now, the signature conversion of a block was only a "partial" IR
rewrite. Rollbacks were triggered via
`BlockTypeConversionRewrite::rollback`, but there was no
`BlockTypeConversionRewrite::commit` equivalent.
Overview of changes:
* Remove `ArgConverter`, an internal helper class that kept track of all
block type conversions. There is now a separate
`BlockTypeConversionRewrite` for each block type conversion.
* No more special handling for block type conversions. They are now
normal "IR rewrites", just like "block creation" or "block movement". In
particular, trigger "commits" of block type conversion via
`BlockTypeConversionRewrite::commit`.
* Remove `ArgConverter::notifyOpRemoved`. This function was used to
inform the `ArgConverter` that an operation was erased, to prevent a
double-free of operations in certain situations. It would be unpractical
to add a `notifyOpRemoved` API to `IRRewrite`. Instead, erasing
ops/block should go through a new `SingleEraseRewriter` (that is owned
by the `ConversionPatternRewriterImpl`) if there is chance of
double-free. This rewriter ignores `eraseOp`/`eraseBlock` if the
op/block was already freed.
This commit improves the block signature conversion API of the dialect
conversion.
There is the following comment in
`ArgConverter::applySignatureConversion`:
```
// If no arguments are being changed or added, there is nothing to do.
```
However, the implementation actually used to replace a block with a new
block even if the block argument types do not change (i.e., there is
"nothing to do"). This is fixed in this commit. The documentation of the
public `ConversionPatternRewriter` API is updated accordingly.
This commit also removes a check that used to *sometimes* skip a block
signature conversion if the block was already converted. This is not
consistent with the public `ConversionPatternRewriter` API; blocks
should always be converted, regardless of whether they were already
converted or not.
Block signature conversion also used to be silently skipped when the
specified block was detached. Instead of silently skipping, an assertion
is triggered. Attempting to convert a detached block (which is likely an
erased block) is invalid API usage.
When a `ModifyOperationRewrite` is committed, the operation may already
have been erased, so `OperationName` must be cached in the rewrite
object.
Note: This will no longer be needed with #81757, which adds a "cleanup"
method to `IRRewrite`.
* When converting a block signature, `ArgConverter` creates a new block
with the new signature and moves all operation from the old block to the
new block. The new block is temporarily inserted into a region that is
stored in `regionMapping`. The old block is not yet deleted, so that the
conversion can be rolled back. `regionMapping` is not needed. Instead of
moving the old block to a temporary region, it can just be unlinked.
Block erasures are handles in the same way in the dialect conversion.
* `regionToConverter` is a mapping from regions to type converter. That
field is never accessed within `ArgConverter`. It should be stored in
`ConversionPatternRewriterImpl` instead.
* `convertedBlocks` is not needed. Old blocks are already stored in
`ConvertedBlockInfo`.
The dialect conversion rolls back in-place op modifications upon
failure. Rolling back modifications of attributes is already supported,
but there was no support for properties until now.
This commit simplifies the internal state of the dialect conversion. A
separate field for the previous state of in-place op modifications is no
longer needed.
Rename listener callback names:
* `notifyOperationRemoved` -> `notifyOperationErased`
* `notifyBlockRemoved` -> `notifyBlockErased`
The current callback names are misnomers. The callbacks are triggered
when an operation/block is erased, not when it is removed (unlinked).
E.g.:
```c++
/// Notify the listener that the specified operation is about to be erased.
/// At this point, the operation has zero uses.
///
/// Note: This notification is not triggered when unlinking an operation.
virtual void notifyOperationErased(Operation *op) {}
```
This change is in preparation of adding listener support to the dialect
conversion. The dialect conversion internally unlinks IR before erasing
it at a later point of time. There is an important difference between
"remove" and "erase". Lister callback names should be accurate to avoid
confusion.
