Add extra error checking (in debug mode) to detect cases where an
iterator on "direct" payload IR mappings is invalidated (due to elements
being removed). Such errors are hard to debug: they are often
non-deterministic; sometimes the program crashes, sometimes it produces
wrong results. Even when it crashes, the stack trace often points to
completely unrelated code locations.
Store a timestamp with each "direct" mapping. The timestamp is increased
whenever an operation is performed that invaldiates an iterator on that
mapping. A debug iterator is added that checks the timestamp as payload
IR is enumerated.
Fixes a crash when an op, that is mapped to handle that a
`transform.foreach` iterates over, was erased (through the
`TrackingRewriter`). Erasing an op removes it from all mappings and
invalidates iterators. This is already taken care of when an op is
iterating over payload ops in its `apply` method, but not when another
transform op is erasing a tracked payload op.
The `MapVector` type stores key-value pairs in a vector, which, when
resized, copies the entries and destroys the old ones. This causes the
underlying operations to be deleted, subsequently causing segfaults.
This patch makes the `mappings` map type refer to a shared pointer
instead, so that resizing the vector doesn't call the operations'
destructors.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D154511
In `TestTensorTransforms.cpp` `replaced` is nullptr I assumed the intent
was to emit the error for the `rootOp`.
In `TransformInterfaces.cpp` there were some uninitialized variables.
In `NVGPUTransformOps.cpp` `matmulOp` was never used.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D154439
When an operation is removed/replaced, the TrackingListener updates the internal transform state mapping between handles and payload IR. All handles must be updated, even the ones that are defined in a region that is beyond the most recent region that is isolated from above.
This fixes a bug, where a payload op was erased in a named sequence. Not only handles defined inside of the named region must be updated, but also all other handles such as the ones where the sequence is included.
Differential Revision: https://reviews.llvm.org/D153767
Do not swap the Mappings when entering a region that is isolated from above. Simply push another Mappings struct to the stack and prevent invalid accesses during lookups.
Differential Revision: https://reviews.llvm.org/D153765
This function allows users to update payload op mappings in cases where such replacements cannot be performed automatically by the rewriter/listener interface.
Differential Revision: https://reviews.llvm.org/D153764
When exiting the scope of a region attached to a transform op, clean up
the handle invalidation checks assocaited with handles defined in this
region. Otherwise, these checks may trigger on the next entry to the
region while there is no incorrect usage.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D153545
All `apply` functions now have a `TransformRewriter &` parameter. This rewriter should be used to modify the IR. It has a `TrackingListener` attached and updates the internal handle-payload mappings based on rewrites.
Implementations no longer need to create their own `TrackingListener` and `IRRewriter`. Error checking is integrated into `applyTransform`. Tracking listener errors are reported only for ops with the `ReportTrackingListenerFailuresOpTrait` trait attached, allowing for a gradual migration. Furthermore, errors can be silenced with an op attribute.
Additional API will be added to `TransformRewriter` in subsequent revisions. This revision just adds an "empty" `TransformRewriter` class and updates all `apply` implementations.
Differential Revision: https://reviews.llvm.org/D152427
As a convenience to the user, top-level sequence ops can optionally be used as matchers: the op type is specified by the type of the block argument.
This is similar to how pass pipeline targets can be specified on the command line (`-pass-pipeline='builtin.module(func.func(...))`).
Differential Revision: https://reviews.llvm.org/D153121
ApplyEachOpTrait applies to payload ops associated with its operand
handle one-by-one in order. If a handle is consumed, this usually
indicates that the associated payload ops are erased or rewritten. Add a
check that we don't consume an ancestor payload operation before
consuming its descendant, as the latter is likely to be a dangling
pointer. Transform operations for which this is a legitimate behavior
(i.e., they consume the handle but don't actually erase or rewrite the
payload operation) should implement the interface directly and allow for
repeated handles.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D152510
A TransformRewriter (with attached TrackingListener) will be added to an interface method in a subsequent revision.
Differential Revision: https://reviews.llvm.org/D152426
All vector transform ops are now `PatternDescriptorOpInterface` ops that merely select the patterns. The patterns are applied by the `apply_patterns` op. This is to ensure that ops are properly tracked. (TrackingListener is used in the implementation of `apply_patterns`.) Furthermore, handles are no longer invalidated when applying patterns in the vector tests.
Differential Revision: https://reviews.llvm.org/D152174
Transform operations may indicate that they may accept and consume
several handles pointing to the same or nested payload entities. The
initial implementation of the expensive-checks mode was simply ignoring
such cases as consuming the second handle would fail the check after the
first handle invalidated it by consuming the same payload. Additional
checks had been added since then, which could now trigger assertions in
the expensive-checks module itself (instead of or in addition to
use-after-free assertions down the road), specifically because the
payload associations for invalidated handles is removed from the state
to enable other kinds of checking.
Rework the handling of transform operations with repeated handles so
use-after-consume is still reported properly if the consumption happened
by a preceding operation, as opposed to the a preceding operand of the
same operation that is still (corretly) ignored if the op requests that.
Depends on: D151560
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D151569
The transform dialect interpreter features the expensive-checks mode
that acts as an embedded sanitizer to track use-after-consume of
transform handles. Its logic is based on the relations between payload
operations, which made it silently ignore empty handles that are
consumed. Also catch and report this case because the remaining code may
hit an assertion on attempting to access a consumed handle (that is
removed from the mapping).
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D151560
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D151542
Certain InsertSliceOps, that do not use elements from the destination, are treated like casts when looking for replacement ops. Such InsertSliceOps are typically rank expansions.
Tensors with dynamic shape are not supported at the moment.
Also adds test cases for the TrackingListener.
Differential Revision: https://reviews.llvm.org/D151422
The initial bring-up of the Transform dialect relied on PDL to provide
the default handle type (`!pdl.operation`) and the matching capability.
Both are now provided natively by the Transform dialect removing the
reason to have a hard dependency on the PDL dialect and its interpreter.
Move PDL-related transform operations into a separate extension.
This requires us to introduce a dialect state extension mechanism into
the Transform dialect so it no longer needs to know about PDL constraint
functions that may be injected by extensions similarly to operations and
types. This mechanism will be reused to connect pattern application
drivers and the Transform dialect.
This completes the restructuring of the Transform dialect to remove
overrilance on PDL.
Note to downstreams: flow that are using `!pdl.operation` with Transform
dialect operations will now require `transform::PDLExtension` to be
applied to the transform dialect in order to provide the transform
handle type interface for `!pdl.operation`.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D151104
Structured fusion proceeds by iteratively finding the next suitable
producer to be fused into the loop. Therefore, it shouldn't matter if
the same producer is listed multiple times (e.g., it is used as multiple
operands). Adjust the implementation of the transform op to support this
case.
Also fix the checking code in the interpreter to actually respect the
TransformOpInterface indication that repeated payload is allowed, it
seems to have been accidentally dropped in one of the refactorings.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D150561
Instead of returning an `ArrayRef<Operation *>`, return at iterator that skips ops that were erased/replaced while iterating over the payload ops.
This fixes an issue in conjuction with TrackingListener, where a tracked op was erased during the iteration. Elements may not be removed from an array while iterating over it; this invalidates the iterator.
When ops are erased/removed via `replacePayloadOp`, they are not immediately removed from the mappings data structure. Instead, they are set to `nullptr`. `nullptr`s are not enumerated by `getPayloadOps`. At the end of each transformation, `nullptr`s are removed from the mapping data structure.
Differential Revision: https://reviews.llvm.org/D149847
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
* https://mlir.llvm.org/deprecation/ at "Use the free function variants for dyn_cast/cast/isa/…"
* Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This follows a previous patch that updated calls
`op.cast<T>()-> cast<T>(op)`. However some cases could not handle an
unprefixed `cast` call due to occurrences of variables named cast, or
occurring inside of class definitions which would resolve to the method.
All C++ files that did not work automatically with `cast<T>()` are
updated here to `llvm::cast` and similar with the intention that they
can be easily updated after the methods are removed through a
find-replace.
See https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
for the clang-tidy check that is used and then update printed
occurrences of the function to include `llvm::` before.
One can then run the following:
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-export-fixes /tmp/cast/casts.yaml mlir/*\
-header-filter=mlir/ -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D150348
Add a set of transform operations into the "structured" extension of the
Transform dialect that allow one to select transformation targets more
specifically than the currently available matching. In particular, add
the mechanism for identifying the producers of operands (input and init
in destination-passing style) and users of results, as well as
mechanisms for reasoning about the shape of the iteration space.
Additionally, add several transform operations to manipulate parameters
that could be useful to implement more advanced selectors. Specifically,
new operations let one produce and compare parameter values to implement
shape-driven transformations.
New operations are placed in separate files to decrease compilation
time. Some relayering of the extension is necessary to avoid repeated
generation of enums.
Depends on D148013
Depends on D148014
Depends on D148015
Reviewed By: chelini
Differential Revision: https://reviews.llvm.org/D148017
Add a new transform op combinator that implements an "if-then-else"
style of mechanism for applying transformations. Its main purpose is to
serve as a higher-level driver when applying multiple transform scripts
to potentially overlapping pieces of the payload IR. This is similar to
how the various rewrite drivers operate in C++, but at a higher level
and with more declarative expressions. This is not intended to replace
existing pattern-based rewrites, but to to drive more complex
transformations that are exposed in the transform dialect and are too
complex to be expressed as simple declarative rewrites.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D148013
This revision adds additional "expensive-checks" checks to the transform dialect that detect the most common cases of:
* Missing `consumesHandle` side effects on transform ops.
* Patterns that remove operations but do not notify the transform dialect.
In essence, these additional checks are looking for dangling pointers to erased payload ops in the transform dialect state and crash the program execution (by dereferencing free'd memory) or triggering an assertion failure. It is recommended to run these extra checks with ASAN. Otherwise, certain failures may not be detected. The ASAN error message can also be used to find the faulty transform op/pattern.
This change also fixes a few faulty transform ops.
Differential Revision: https://reviews.llvm.org/D147447
In particular, move the printing of the top-level payload after each
transform under the "full output" debug flag, it is rarely useful and
excessively long. Also don't print the regions of the transform
operation being applied as each individual operation in the region is
likely going to be applied later by itself and therefore printed.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D148014
Use the argument attribute mechanism for function-like operations to
annotate the arguments of named transform sequences as consuming or only
reading the handles passed as arguments. This makes it possible to
correctly specify handle invalidation for external named sequences by
requiring their declarations to always provide such annotations.
Additionally, these annotations remove the need to analyze the body of
a named sequence to understand its effects on the arguments. Make them
required for named sequences that are called from the same file, in
addition to external sequences.
Provide a convenience pass that infers annotations by analyzing bodies
of named sequences provided they are not called from the same file.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D147223
This revision adds vector transform operations that allow us to better inspect the composition
of various lowerings that were previously very opaque.
This commit is NFC in that it does not change patterns beyond adding `rewriter.notifyFailure` messages
and it does not change the tests beyond breaking them into pieces and using transforms instead of
throwaway opaque test passes.
Reviewed By: ftynse, springerm
Co-authored-by: Alex Zinenko <zinenko@google.com>
Differential Revision: https://reviews.llvm.org/D146755
Named sequences introduce an additional abstraction and reuse capability
to the transform dialect. They can be though of as macros parameterized
with handles that can be invoked in places where a transform dialect
operation is expected. Such reuse was previously not possible in the
dialect and required dynamic construction of the transform IR from the
client language. Named sequences are intentionally restricted to
disallow recursion, as it could make the dialect accidentally
Turing-complete, which isn't desired at this point.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D146433
Replace references to enumerate results with either result_pairs
(reference wrapper type) or structured bindings. I did not use
structured bindings everywhere as it wasn't clear to me it would
improve readability.
This is in preparation to the switch to zip semantics which won't
support non-const lvalue reference to elements:
https://reviews.llvm.org/D144503.
I chose to use values instead of const lvalue-refs because MLIR is
biased towards avoiding `const` local variables. This won't degrade
performance because currently `result_pair` is cheap to copy (size_t
+ iterator), and in the future, the enumerator iterator dereference
will return temporaries anyway.
Reviewed By: dblaikie
Differential Revision: https://reviews.llvm.org/D146006
The initial implementaiton of value handle update when the payload
operation defining the values associated with value handles was being
replaced required the replacement operation to have the same number of
results. This is not strictly necessary. The replacement operation may
have more results, or less results provided that there are no handles to
the results that have no equivalent in the replacement op.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D145254
This transform op returns a value handle pointing to the specified OpResult of the targeted op.
Differential Revision: https://reviews.llvm.org/D144087
Previous changes in 98acd74683 were overly
eager to disallow null payload everywhere. The semantics of
TransformEachOpTrait allows individual applications to return null
payloads as means of filtering out the operations to which they are not
applicable without emitting even a silenceable failure. This is a
questionable choice, but one apparently relied upon. Null payloads are
not supposed to leak outside of the trait.
Reviewed By: qcolombet
Differential Revision: https://reviews.llvm.org/D143904
Introduce support for the third kind of values in the transform dialect:
value handles. Similarly to operation handles, value handles are
pointing to a set of values in the payload IR. This enables
transformation to be targeted at specific values, such as individual
results of a multi-result payload operation without indirecting through
the producing op or block arguments that previously could not be easily
addressed. This is expected to support a broad class of memory-oriented
transformations such as selective bufferization, buffer assignment, and
memory transfer management.
Value handles are functionally similar to operation handles and require
similar implementation logic. The most important change concerns the
handle invalidation mechanism where operation and value handles can
affect each other.
This patch includes two cleanups that make it easier to introduce value
handles:
- `RaggedArray` structure that encapsulates the SmallVector of
ArrayRef backed by flat SmallVector logic, frequently used in the
transform interfaces implementation;
- rewrite the tests that associated payload handles with an integer
value `reinterpret_cast`ed as a pointer, which were a frequent
source of confusion and crashes when adding more debugging
facilities that can inspect the payload.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D143385
Add a verifier checking that if a transform operation consumes a handle
(which is associated with a payload operation being erased or
recreated), it also indicates modification of the payload IR. This
hasn't been consistent in the past because of the "no-aliasing"
assumption where we couldn't have had more than one handle to an
operation, requiring some handle-manipulation operations, such as
`transform.merge_handles` to consume their operands. That assumption has
been liften and it is no longer necessary for these operations to
consume handles and thus make the life harder for the clients.
Additionally, remove TransformEffects.td that uses the ODS mechanism for
indicating side effects that works only for operands and results. It
was being used incorrectly to also indicate effects on the payload IR,
not assocaited with any IR value, and lacked the consume/produce
semantics available via helpers in C++.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D142361
The original implementation of the transform interpreter pass base was
cloning the entire transform IR in presence of PDL-related operations to
avoid concurrency issues when running the pass with the same transform
IR on multiple operations of the payload IR. The root cause of those
issues is the `transform.pdl_match` operation that was moving the PDL
pattern definition operation into a new module, consumed by the PDL
interpreter and leading to a race. Clone the pattern operation instead.
This avoids the race as well as the cost for transform IR that doesn't
use PDL.
Depends on D142729.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D142962
The transform dialect infrastructure does not provide a default
interpreter pass and instead expects users to create their own to ensure
all relevant extensions and dependent dialects are loaded. Provide a
base class for implementing such passes that includes the additional
facilities for debugging and is aware of the multithreaded nature of
pass execution.
Reviewed By: pifon2a, nicolasvasilache
Differential Revision: https://reviews.llvm.org/D142729
`applyTransforms` now takes an optional mapping to be associated with
trailing block arguments of the top-level transform op, in addition to
the payload root. This allows for more advanced forms of communication
between C++ code and the transform dialect interpreter, in particular
supplying operations without having to re-match them during
interpretation.
Reviewed By: shabalin
Differential Revision: https://reviews.llvm.org/D142559
Simplify the handling of silenceable failures in the transform dialect.
Previously, the logic of `TransformEachOpTrait` required that
`applyToEach` returned a list of null pointers when a silenceable
failure was emitted. This was not done consistently and also crept into
ops without this trait although they did not require it. Handle this
case earlier in the interpreter and homogeneously associated preivously
unset transform dialect values (both handles and parameters) with empty
lists of the matching kind. Ignore the results of `applyToEach` for the
targets for which it produced a silenceable failure. As a result, one
never needs to set results to lists containing nulls. Furthermore, the
objects associated with transform dialect values must never be null.
Depends On D140980
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D141305
Add a verifier to the TransformOpInterface ensuring that operations
implementing the interface define memory effects on their operands and
results.
Add the missing effects to TileToForeachThreadOp, specifically for
operands that were added at a later version of the op without modifying
`getEffects` accordingly.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D141371
When establishing the correspondence between transform values and
payload operations or parameters, check that the latter are non-null and
report errors. This was previously allowed for exotic cases of partially
successfull transformations with "apply each" trait, but was dangerous.
The "apply each" implementation was reworked to remove the need for this
functionality, so this can now be hardned to avoid null pointer
dereferences.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D141142
Adapt the implementation of TransformEachOpTrait to the existence of
parameter values recently introduced into the transform dialect. In
particular, allow `applyToOne` hooks to return a list containing a mix
of `Operation *` that will be associated with handles and `Attribute`
that will be associated with parameter values by the trait
implementation of the transform interface's `apply` method.
Disentangle the "transposition" of the list of per-payload op partial
results to decrease its overall complexity and detemplatize the code
that doesn't really need templates. This removes the poorly documented
special handling for single-result ops with TransformEachOpTrait that
could have assigned null pointer values to handles.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D140979
