This revision adds support for unstructured control flow to the bufferization infrastructure. In particular: regions with multiple blocks, `cf.br`, `cf.cond_br`.
Two helper templates are added to `BufferizableOpInterface.h`, which can be implemented by ops that supported unstructured control flow in their regions (e.g., `func.func`) and ops that branch to another block (e.g., `cf.br`).
A block signature is always bufferized together with the op that owns the block.
Differential Revision: https://reviews.llvm.org/D158094
Moves the lowering of `bufferization.dealloc` to memref into a separate pass,
but still registers the pattern in the conversion pass. This is helpful when
some tensor values (and thus `to_memref` or `to_tensor` operations) still
remain, e.g., when the function boundaries are not converted, or when constant
tensors are converted to memref.get_global at a later point.
However, it is still recommended to perform all bufferization before
deallocation to avoid memory leaks as all memref allocations inserted after the
deallocation pass was applied, have to be handled manually.
Note: The buffer deallocation pass assumes that memref values defined by
`bufferization.to_memref` don't return ownership and don't have to be
deallocated. `bufferization.to_tensor` operations are handled similarly to
`bufferization.clone` operations with the exception that the result value is
not handled because it's a tensor (not a memref).
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D159180
This new pattern allows us to simplify the dealloc result value (by replacing
it with a constant 'true') and to trim the 'memref' operand list when we know
that all retained memrefs alias with one in the 'memref' list that has a
constant 'true' condition. Because the conditions of aliasing memrefs are
combined by disjunction, we know that once a single constant 'true' value is in
the disjunction the remaining elements don't matter anymore. This complements
the RemoveDeallocMemrefsContainedInRetained pattern which removes values from
the 'memref' list when static information is available for all retained values
by also allowing to remove values in the presence of may-aliases, but under
above mentioned condition instead.
The BufferDeallocation pass often adds dealloc operations where the memref and
retain lists are the same and all conditions are 'true'. If the operands are
all function arguments, for example, they are always determined to may-alias
which renders the other patterns invalid, but the op could still be trivially
optimized away. It would even be enough to directly compare the two operand
lists and check the conditions are all constant 'true' (plus checking for the
extract_strided_metadata operation), but this pattern is a bit more general and
still works when there are additional memrefs in the 'memref' list that actually
have to be deallocated (e.g., see regression test).
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D158518
We are allowed to remove any values from the `memref` list for which there is no
memref in the `retained` list with a may-alias relation. Before removing, we
just have to make sure that the corresponding op results for all retained
memrefs with must-alias relation are updated accordingly. This means, the the
condition operand has to become part of the disjunction the result value is
computed with.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D158395
This revision is needed to support bufferization of `cf.br`/`cf.cond_br`. It will also be useful for better analysis of loop ops.
This revision generalizes `getAliasingOpResults` to `getAliasingValues`. An OpOperand can now not only alias with OpResults but also with BlockArguments. In the case of `cf.br` (will be added in a later revision): a `cf.br` operand will alias with the corresponding argument of the destination block.
If an op does not implement the `BufferizableOpInterface`, the analysis in conservative. It previously assumed that an OpOperand may alias with each OpResult. It now assumes that an OpOperand may alias with each OpResult and each BlockArgument of the entry block.
Differential Revision: https://reviews.llvm.org/D157957
Add a pattern that splits one dealloc operation into multiple dealloc operation
depending on static aliasing information of the values in the `memref` operand
list. This reduces the total number of aliasing checks required at runtime and
can enable futher canonicalizations of the new and simplified dealloc
operations.
Depends on D157407
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D157508
Adds a pattern that removes memrefs from the `retained` list which are
guaranteed to not alias any memref in the `memrefs` list. The corresponding
result value can be replaced with `false` in that case according to the
operation description.
When applied after BufferDeallocation, this can considerably reduce the
overhead that needs to be added during the lowering of the dealloc operation to
check for aliasing (especially when there is only one element in the `memref`
list and all `retained` values can be removed).
Depends on D157398
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D157407
Adds a pass that can be run after buffer deallocation to simplify the deallocation operations.
In particular, there are patterns that need alias information and thus cannot be added as a regular canonicalization pattern.
This initial commit moves an incorrect canonicalization pattern from over to this new pass and fixes it by querying the alias analysis for the additional information it needs to be correct (there must not by any potential aliasing memref in the retain list other than the currently mached one).
Also, improves this pattern by considering the `extract_strided_metadata` operation which is inserted by the deallocation pass by default.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D157398
EmptyTensorElimination is a pre-bufferization transformation that replaces "tensor.empty" ops with "tensor.extract_slice" ops. This revision adds support for cases where the input IR contains "tensor.cast" ops.
Differential Revision: https://reviews.llvm.org/D156167
To keep the pass simple, users should apply cleanup passes manually when necessary. In particular, `-cse -canonicalize` are often desireable to fold away self-copies that are created by the bufferization.
This addresses a comment in D120191.
Differential Revision: https://reviews.llvm.org/D155923
This unit attribute indicates to the bufferization that the resulting buffer will not be written to by another op.
Differential Revision: https://reviews.llvm.org/D154967
This operation is a "copy" operation on tensors. It is guaranteed to bufferize to a memcpy. This is different from "tensor.insert_slice", which may fold away.
Note: There is a symmetry between certain tensor, bufferization and memref ops:
* `tensor.empty`, `bufferization.alloc_tensor`, `memref.alloc`
* (none), `bufferization.dealloc_tensor`, `memref.dealloc`
* `tensor.insert_slice`, `bufferization.copy_tensor`, `memref.copy`
Tensor ops can generally canonicalize/fold away, while bufferization dialect ops can be used when a certain side effect is expected to materialize; so they do not fold away.
Differential Revision: https://reviews.llvm.org/D153552
bufferization.to_memref ops are allowed in One-Shot Bufferize, but they are treated conservatively: in the absence of a memref analysis, we have to assume that the result buffer is read and written.
Note: to_memref cannot introduce any future aliases that would have to be considered during One-Shot Bufferize, because only to_tensor ops with the `restrict` attribute are supported. Such tensors are guaranteed to not alias with any other buffer after bufferization.
Differential Revision: https://reviews.llvm.org/D153365
Update operations in Transform dialect extensions defined in the Affine,
GPU, MemRef and Tensor dialects to use the more generic
`TransformHandleTypeInterface` type constraint instead of hardcoding
`PDL_Operation`. See
https://discourse.llvm.org/t/rfc-type-system-for-the-transform-dialect/65702
for motivation.
Remove the dependency on PDLDialect from these extensions.
Update tests to use `!transform.any_op` instead of `!pdl.operation`.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D150781
Block arguments and yielded values are not equivalent if there are not enough block arguments. This fixes#59442.
Differential Revision: https://reviews.llvm.org/D145575
If an op is unknown to the analysis, it must be treated conservatively: assume that every operand aliases with every result.
Differential Revision: https://reviews.llvm.org/D150546
Restrict the op to functions and modules. Such ops are modified in-place. The transform now consumes the handle and produces a new handle. The `target_is_module` attribute is no longer needed because a result handle is produced in either case.
Differential Revision: https://reviews.llvm.org/D147446
`restrict` is similar to the C++ restrict keyword. Results of `to_tensor` that have the `restrict` attribute are guaranteed to not alias any other `to_tensor` result (after bufferization).
Note: Since `to_memref` ops are not supported by One-Shot Bufferize and all bufferizable ops follow DPS rules (i.e., the buffer of the result is the buffer of an operand or an alias thereof), the buffer of a `to_tensor` op that has the `restrict` attribute is always an entirely "new" buffer that is not aliasing with the future buffer of any tensor value in the entire program. This makes such `to_tensor` ops "safe" from a bufferization perspective; they cannot cause RaW conflicts.
Differential Revision: https://reviews.llvm.org/D144021
The current bufferization on function boundaries works on `func.func`
and any call op implementing `CallOpInterface`. Then, an error is thrown
if there is a `CallOpInterface` op that is not `func.call`. This is
unnecessary and breaks the pass whenever such an op occurs (such as
`llvm.call`). This PR simply restricts the handling of call ops to
`func.call`.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D143724
Reading from tensor.empty or bufferization.alloc_tensor (without copy) cannot cause a conflict because these ops do not specify the contents of their result tensors.
Differential Revision: https://reviews.llvm.org/D143183
Checks were too strict and by the time the patch was submitted,
the output of the test changed.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D142969
OneShotModuleBufferize fails if the input IR cannot be analyzed.
One can set CopyBeforeWrite=true in order to skip analysis.
In that case, a buffer copy is inserted on every write.
This leads to many copies, also in FuncOps that could be analyzed.
This change aims to copy buffers only when it is a must.
When running OneShotModuleBufferize with CopyBeforeWrite=false,
FuncOps whose names are specified in noAnalysisFuncFilter will not be
analyzed. Ops in these FuncOps will not be analyzed as well.
They will be bufferized with CopyBeforeWrite=true,
while the other ops will be bufferized with CopyBeforeWrite=false.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D142631
Unranked tensors can currently not be copied. They are forced to always bufferize in-place. There is typically some other OpOperand that can bufferize out-of-place instead if needed.
Note: There is IR that cannot be bufferized with One-Shot Bufferize at the moment (see invalid test case). But it is unclear if we need to support such cases. We do not have a use case at the moment. This restriction could be loosened in the future if needed.
This change improves error handling when bufferizing IR where an unranked tensor would be copied. It also disables an optimization where an OpResult was copied instead of an OpOperand in case the OpResult is an unranked tensor (Github #60187).
Differential Revision: https://reviews.llvm.org/D142331
This allows much better verification messages in consuming ops that properly declare
`TransformHandleTypeInterface` on their operands.
Downstream tests can be updated with a command resembling:
```
git grep -l "structured\.match" mlir/test | xargs -i sed -i {} -e "s/\(structured.match.*\)/\1 : (\!pdl.operation) -> \!pdl.operation/g"
```
Differential Revision: https://reviews.llvm.org/D142643
Introduce a new transform operation to replace `tensor.empty` with
`alloc_tensor` operations. The operation is a pass-through if the target
operation is already a `alloc_tensor`; otherwise, it expects a
`tensor.empty` as a target. Currently, it does not return any results.
The operation is expected to run before `one_shot_bufferize` as
`one_shot_bufferize` rejects `tensor.empty`.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D140026
External functions have no body, so they cannot be analyzed. Assume conservatively that each tensor bbArg may be aliasing with each tensor result. Furthermore, assume that each function arg is read and written-to after bufferization. This default behavior can be controlled with `bufferization.access` (similar to `bufferization.memory_layout`) in test cases.
Also fix a bug in the dialect attribute verifier, which did not run for region argument attributes.
Differential Revision: https://reviews.llvm.org/D139517
While it is unlikely to matter in practice, there is no reason
for this value to be larger than it should be. 64 bytes is the
size of a cache line in most machines, and we can fit a full
512-bit vector in it.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D139434
TensorCopyInsertion should not have been exposed as a pass. This was a flaw in the original design. It is a preparation step for bufferization and certain transforms (that would otherwise be legal) are illegal between TensorCopyInsertion and actual rewrite to MemRef ops. Therefore, even if broken down as two separate steps internally, they should be exposed as a single pass.
This change affects the sparse compiler, which uses `TensorCopyInsertionPass`. A new `SparsificationAndBufferizationPass` is added to replace all passes in the sparse tensor pipeline from `TensorCopyInsertionPass` until the actual bufferization (rewrite to memref/non-tensor). It is generally unsafe to run arbitrary passes in-between, in particular passes that hoist tensor ops out of loops or change SSA use-def chains along tensor ops.
Differential Revision: https://reviews.llvm.org/D138915
The previous error message was confusing. Also improve code documentation and some minor code cleanups.
Differential Revision: https://reviews.llvm.org/D138902
Adds RegionBranchOpInterface for AffineIf Op and tests it
using buffer deallocation pass.
Reviewed By: bondhugula
Differential Revision: https://reviews.llvm.org/D130962
MemRef has been accepting a general Attribute as memory space for
a long time. This commits updates bufferization side to catch up,
which allows downstream users to plugin customized symbolic memory
space. This also eliminates quite a few `getMemorySpaceAsInt`
calls, which is deprecated.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D138330
Expose `function-boundary-type-conversion` in `OneShotBufferizeOp`. To
reuse options between passes and transform operations, create a
`BufferizationEnums.td`.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D137833
