The tracking listener should not report op replacement errors for
payload ops that are not mapped to any live handles. The handle liveless
analysis did not work properly with transform IR that has named
sequences.
A handle is live if it has a user after the transform op that is
currently being applied. With named sequences, we need to maintain a
stack of currently applied transform ops. That stack already exists
(`regionStack`), the only thing that's missing is the current transform
op for each stack frame.
This commit fixes#72931.
This check was trying to find cases of invalid API usage:
incorrect/missing handle side effects and/or incorrect rewriter usage.
This check is not implemented correctly and can report false positives
in case of pointer reuse (different op created at same location). It is
unclear if such a check can be implemented given that we have both
tracking listener-based handle updates and handle consumption.
Fixes#72931.
C++20 comes with std::erase to erase a value from std::vector. This
patch renames llvm::erase_value to llvm::erase for consistency with
C++20.
We could make llvm::erase more similar to std::erase by having it
return the number of elements removed, but I'm not doing that for now
because nobody seems to care about that in our code base.
Since there are only 50 occurrences of erase_value in our code base,
this patch replaces all of them with llvm::erase and deprecates
llvm::erase_value.
This revision provides the ability to use an arbitrary named sequence op
as
the entry point to a transform dialect strategy.
It is also a step towards better transform dialect usage in pass
pipelines
that need to preload a transform library rather thanparse it on the fly.
The interpreter itself is significantly simpler than its testing
counterpart
by avoiding payload/debug root tags and multiple shared modules.
In the process, the NamedSequenceOp::apply function is adapted to allow
it
being an entry point.
NamedSequenceOp is **not** extended to take the PossibleTopLevelTrait at
this
time, because the implementation of the trait is specific to allowing
one
top-level dangling op with a region such as SequenceOp or
AlternativesOp.
In particular, the verifier of PossibleTopLevelTrait does not allow for
an
empty body, which is necessary to declare a NamedSequenceOp that gets
linked
in separately before application.
In the future, we should dispense with the PossibleTopLevelTrait
altogether
and always enter the interpreter with a NamedSequenceOp.
Lastly, relevant TD linking utilities are moved to
TransformInterpreterUtils
and reused from there.
Fix a crash when consuming an op in a named sequence, when the same op
is also mapped in the caller's mapping. Ops must be removed from *all*
mappings during the "expensive checks". Otherwise, we may have dangling
pointers in the mappings data structures, which interfere with the
expensive checks.
This commit reapplies #66987, which got original contained a memory leak
and got reverted by 78c8ab5844. The leak
is now fixed.
Original description:
This PR extends the error message of the tracking listener when
replacement ops cannot be found. That may happen if the applied patterns
replace an op by an op of a different kind or by block arguments.
However, this only matters if there are alive handles to the replaced
op. The new error message mentions that explicitly and reports the alive
handles.
Same as #66369 but for payload values. (#66369 added checks only for
payload operations.)
It was necessary to change the signature of `getPayloadValues` to return
an iterator. This is now similar to payload operations.
Fixes an issue in #66369 where the `LLVM_ENABLE_ABI_BREAKING_CHECKS`
check was inverted.
This PR extends the error message of the tracking listener when
replacement ops cannot be found. That may happen if the applied patterns
replace an op by an op of a different kind or by block arguments.
However, this only matters if there are alive handles to the replaced
op. The new error message mentions that explicitly and reports the alive
handles.
Add extra error checking (in debug mode) to detect cases where an
iterator on "direct" payload IR mappings is invalidated (due to elements
being removed). Such errors are hard to debug: they are often
non-deterministic; sometimes the program crashes, sometimes it produces
wrong results. Even when it crashes, the stack trace often points to
completely unrelated code locations.
Store a timestamp with each "direct" mapping. The timestamp is increased
whenever an operation is performed that invaldiates an iterator on that
mapping. A debug iterator is added that checks the timestamp as payload
IR is enumerated.
Fixes a crash when an op, that is mapped to handle that a
`transform.foreach` iterates over, was erased (through the
`TrackingRewriter`). Erasing an op removes it from all mappings and
invalidates iterators. This is already taken care of when an op is
iterating over payload ops in its `apply` method, but not when another
transform op is erasing a tracked payload op.
The `MapVector` type stores key-value pairs in a vector, which, when
resized, copies the entries and destroys the old ones. This causes the
underlying operations to be deleted, subsequently causing segfaults.
This patch makes the `mappings` map type refer to a shared pointer
instead, so that resizing the vector doesn't call the operations'
destructors.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D154511
In `TestTensorTransforms.cpp` `replaced` is nullptr I assumed the intent
was to emit the error for the `rootOp`.
In `TransformInterfaces.cpp` there were some uninitialized variables.
In `NVGPUTransformOps.cpp` `matmulOp` was never used.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D154439
When an operation is removed/replaced, the TrackingListener updates the internal transform state mapping between handles and payload IR. All handles must be updated, even the ones that are defined in a region that is beyond the most recent region that is isolated from above.
This fixes a bug, where a payload op was erased in a named sequence. Not only handles defined inside of the named region must be updated, but also all other handles such as the ones where the sequence is included.
Differential Revision: https://reviews.llvm.org/D153767
Do not swap the Mappings when entering a region that is isolated from above. Simply push another Mappings struct to the stack and prevent invalid accesses during lookups.
Differential Revision: https://reviews.llvm.org/D153765
This function allows users to update payload op mappings in cases where such replacements cannot be performed automatically by the rewriter/listener interface.
Differential Revision: https://reviews.llvm.org/D153764
When exiting the scope of a region attached to a transform op, clean up
the handle invalidation checks assocaited with handles defined in this
region. Otherwise, these checks may trigger on the next entry to the
region while there is no incorrect usage.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D153545
All `apply` functions now have a `TransformRewriter &` parameter. This rewriter should be used to modify the IR. It has a `TrackingListener` attached and updates the internal handle-payload mappings based on rewrites.
Implementations no longer need to create their own `TrackingListener` and `IRRewriter`. Error checking is integrated into `applyTransform`. Tracking listener errors are reported only for ops with the `ReportTrackingListenerFailuresOpTrait` trait attached, allowing for a gradual migration. Furthermore, errors can be silenced with an op attribute.
Additional API will be added to `TransformRewriter` in subsequent revisions. This revision just adds an "empty" `TransformRewriter` class and updates all `apply` implementations.
Differential Revision: https://reviews.llvm.org/D152427
As a convenience to the user, top-level sequence ops can optionally be used as matchers: the op type is specified by the type of the block argument.
This is similar to how pass pipeline targets can be specified on the command line (`-pass-pipeline='builtin.module(func.func(...))`).
Differential Revision: https://reviews.llvm.org/D153121
ApplyEachOpTrait applies to payload ops associated with its operand
handle one-by-one in order. If a handle is consumed, this usually
indicates that the associated payload ops are erased or rewritten. Add a
check that we don't consume an ancestor payload operation before
consuming its descendant, as the latter is likely to be a dangling
pointer. Transform operations for which this is a legitimate behavior
(i.e., they consume the handle but don't actually erase or rewrite the
payload operation) should implement the interface directly and allow for
repeated handles.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D152510
A TransformRewriter (with attached TrackingListener) will be added to an interface method in a subsequent revision.
Differential Revision: https://reviews.llvm.org/D152426
All vector transform ops are now `PatternDescriptorOpInterface` ops that merely select the patterns. The patterns are applied by the `apply_patterns` op. This is to ensure that ops are properly tracked. (TrackingListener is used in the implementation of `apply_patterns`.) Furthermore, handles are no longer invalidated when applying patterns in the vector tests.
Differential Revision: https://reviews.llvm.org/D152174
Transform operations may indicate that they may accept and consume
several handles pointing to the same or nested payload entities. The
initial implementation of the expensive-checks mode was simply ignoring
such cases as consuming the second handle would fail the check after the
first handle invalidated it by consuming the same payload. Additional
checks had been added since then, which could now trigger assertions in
the expensive-checks module itself (instead of or in addition to
use-after-free assertions down the road), specifically because the
payload associations for invalidated handles is removed from the state
to enable other kinds of checking.
Rework the handling of transform operations with repeated handles so
use-after-consume is still reported properly if the consumption happened
by a preceding operation, as opposed to the a preceding operand of the
same operation that is still (corretly) ignored if the op requests that.
Depends on: D151560
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D151569
The transform dialect interpreter features the expensive-checks mode
that acts as an embedded sanitizer to track use-after-consume of
transform handles. Its logic is based on the relations between payload
operations, which made it silently ignore empty handles that are
consumed. Also catch and report this case because the remaining code may
hit an assertion on attempting to access a consumed handle (that is
removed from the mapping).
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D151560
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D151542
Certain InsertSliceOps, that do not use elements from the destination, are treated like casts when looking for replacement ops. Such InsertSliceOps are typically rank expansions.
Tensors with dynamic shape are not supported at the moment.
Also adds test cases for the TrackingListener.
Differential Revision: https://reviews.llvm.org/D151422
The initial bring-up of the Transform dialect relied on PDL to provide
the default handle type (`!pdl.operation`) and the matching capability.
Both are now provided natively by the Transform dialect removing the
reason to have a hard dependency on the PDL dialect and its interpreter.
Move PDL-related transform operations into a separate extension.
This requires us to introduce a dialect state extension mechanism into
the Transform dialect so it no longer needs to know about PDL constraint
functions that may be injected by extensions similarly to operations and
types. This mechanism will be reused to connect pattern application
drivers and the Transform dialect.
This completes the restructuring of the Transform dialect to remove
overrilance on PDL.
Note to downstreams: flow that are using `!pdl.operation` with Transform
dialect operations will now require `transform::PDLExtension` to be
applied to the transform dialect in order to provide the transform
handle type interface for `!pdl.operation`.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D151104
Structured fusion proceeds by iteratively finding the next suitable
producer to be fused into the loop. Therefore, it shouldn't matter if
the same producer is listed multiple times (e.g., it is used as multiple
operands). Adjust the implementation of the transform op to support this
case.
Also fix the checking code in the interpreter to actually respect the
TransformOpInterface indication that repeated payload is allowed, it
seems to have been accidentally dropped in one of the refactorings.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D150561
Instead of returning an `ArrayRef<Operation *>`, return at iterator that skips ops that were erased/replaced while iterating over the payload ops.
This fixes an issue in conjuction with TrackingListener, where a tracked op was erased during the iteration. Elements may not be removed from an array while iterating over it; this invalidates the iterator.
When ops are erased/removed via `replacePayloadOp`, they are not immediately removed from the mappings data structure. Instead, they are set to `nullptr`. `nullptr`s are not enumerated by `getPayloadOps`. At the end of each transformation, `nullptr`s are removed from the mapping data structure.
Differential Revision: https://reviews.llvm.org/D149847
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
* https://mlir.llvm.org/deprecation/ at "Use the free function variants for dyn_cast/cast/isa/…"
* Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This follows a previous patch that updated calls
`op.cast<T>()-> cast<T>(op)`. However some cases could not handle an
unprefixed `cast` call due to occurrences of variables named cast, or
occurring inside of class definitions which would resolve to the method.
All C++ files that did not work automatically with `cast<T>()` are
updated here to `llvm::cast` and similar with the intention that they
can be easily updated after the methods are removed through a
find-replace.
See https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
for the clang-tidy check that is used and then update printed
occurrences of the function to include `llvm::` before.
One can then run the following:
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-export-fixes /tmp/cast/casts.yaml mlir/*\
-header-filter=mlir/ -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D150348
Add a set of transform operations into the "structured" extension of the
Transform dialect that allow one to select transformation targets more
specifically than the currently available matching. In particular, add
the mechanism for identifying the producers of operands (input and init
in destination-passing style) and users of results, as well as
mechanisms for reasoning about the shape of the iteration space.
Additionally, add several transform operations to manipulate parameters
that could be useful to implement more advanced selectors. Specifically,
new operations let one produce and compare parameter values to implement
shape-driven transformations.
New operations are placed in separate files to decrease compilation
time. Some relayering of the extension is necessary to avoid repeated
generation of enums.
Depends on D148013
Depends on D148014
Depends on D148015
Reviewed By: chelini
Differential Revision: https://reviews.llvm.org/D148017
Add a new transform op combinator that implements an "if-then-else"
style of mechanism for applying transformations. Its main purpose is to
serve as a higher-level driver when applying multiple transform scripts
to potentially overlapping pieces of the payload IR. This is similar to
how the various rewrite drivers operate in C++, but at a higher level
and with more declarative expressions. This is not intended to replace
existing pattern-based rewrites, but to to drive more complex
transformations that are exposed in the transform dialect and are too
complex to be expressed as simple declarative rewrites.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D148013
This revision adds additional "expensive-checks" checks to the transform dialect that detect the most common cases of:
* Missing `consumesHandle` side effects on transform ops.
* Patterns that remove operations but do not notify the transform dialect.
In essence, these additional checks are looking for dangling pointers to erased payload ops in the transform dialect state and crash the program execution (by dereferencing free'd memory) or triggering an assertion failure. It is recommended to run these extra checks with ASAN. Otherwise, certain failures may not be detected. The ASAN error message can also be used to find the faulty transform op/pattern.
This change also fixes a few faulty transform ops.
Differential Revision: https://reviews.llvm.org/D147447
In particular, move the printing of the top-level payload after each
transform under the "full output" debug flag, it is rarely useful and
excessively long. Also don't print the regions of the transform
operation being applied as each individual operation in the region is
likely going to be applied later by itself and therefore printed.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D148014
Use the argument attribute mechanism for function-like operations to
annotate the arguments of named transform sequences as consuming or only
reading the handles passed as arguments. This makes it possible to
correctly specify handle invalidation for external named sequences by
requiring their declarations to always provide such annotations.
Additionally, these annotations remove the need to analyze the body of
a named sequence to understand its effects on the arguments. Make them
required for named sequences that are called from the same file, in
addition to external sequences.
Provide a convenience pass that infers annotations by analyzing bodies
of named sequences provided they are not called from the same file.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D147223
This revision adds vector transform operations that allow us to better inspect the composition
of various lowerings that were previously very opaque.
This commit is NFC in that it does not change patterns beyond adding `rewriter.notifyFailure` messages
and it does not change the tests beyond breaking them into pieces and using transforms instead of
throwaway opaque test passes.
Reviewed By: ftynse, springerm
Co-authored-by: Alex Zinenko <zinenko@google.com>
Differential Revision: https://reviews.llvm.org/D146755
Named sequences introduce an additional abstraction and reuse capability
to the transform dialect. They can be though of as macros parameterized
with handles that can be invoked in places where a transform dialect
operation is expected. Such reuse was previously not possible in the
dialect and required dynamic construction of the transform IR from the
client language. Named sequences are intentionally restricted to
disallow recursion, as it could make the dialect accidentally
Turing-complete, which isn't desired at this point.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D146433
Replace references to enumerate results with either result_pairs
(reference wrapper type) or structured bindings. I did not use
structured bindings everywhere as it wasn't clear to me it would
improve readability.
This is in preparation to the switch to zip semantics which won't
support non-const lvalue reference to elements:
https://reviews.llvm.org/D144503.
I chose to use values instead of const lvalue-refs because MLIR is
biased towards avoiding `const` local variables. This won't degrade
performance because currently `result_pair` is cheap to copy (size_t
+ iterator), and in the future, the enumerator iterator dereference
will return temporaries anyway.
Reviewed By: dblaikie
Differential Revision: https://reviews.llvm.org/D146006
The initial implementaiton of value handle update when the payload
operation defining the values associated with value handles was being
replaced required the replacement operation to have the same number of
results. This is not strictly necessary. The replacement operation may
have more results, or less results provided that there are no handles to
the results that have no equivalent in the replacement op.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D145254
This transform op returns a value handle pointing to the specified OpResult of the targeted op.
Differential Revision: https://reviews.llvm.org/D144087
Previous changes in 98acd74683 were overly
eager to disallow null payload everywhere. The semantics of
TransformEachOpTrait allows individual applications to return null
payloads as means of filtering out the operations to which they are not
applicable without emitting even a silenceable failure. This is a
questionable choice, but one apparently relied upon. Null payloads are
not supposed to leak outside of the trait.
Reviewed By: qcolombet
Differential Revision: https://reviews.llvm.org/D143904
Introduce support for the third kind of values in the transform dialect:
value handles. Similarly to operation handles, value handles are
pointing to a set of values in the payload IR. This enables
transformation to be targeted at specific values, such as individual
results of a multi-result payload operation without indirecting through
the producing op or block arguments that previously could not be easily
addressed. This is expected to support a broad class of memory-oriented
transformations such as selective bufferization, buffer assignment, and
memory transfer management.
Value handles are functionally similar to operation handles and require
similar implementation logic. The most important change concerns the
handle invalidation mechanism where operation and value handles can
affect each other.
This patch includes two cleanups that make it easier to introduce value
handles:
- `RaggedArray` structure that encapsulates the SmallVector of
ArrayRef backed by flat SmallVector logic, frequently used in the
transform interfaces implementation;
- rewrite the tests that associated payload handles with an integer
value `reinterpret_cast`ed as a pointer, which were a frequent
source of confusion and crashes when adding more debugging
facilities that can inspect the payload.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D143385
