When I landed the EmulateUnsupportedFloats, I'd negligently included
an assert that needed to run for the pass to be correct. Previous
emergency fix commits removed the assert. This commit re-adds the
"can't happen" testing as an emitOpError() and aborting the rewrite,
thus allowing it to function in no-assertions builds.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D155088
asserts get compiled to empty when built in opt mode, so that makes
certain tests fail, such as emulate-unsupported-floats.mlir.test.
This removes the assert altogether, which is also suboptimal, but
I have reported to the original author.
To complement the bf16 expansion and truncation patterns added to
ExpandOps, define a pass that replaces, for any arithmetic operation
op,
%y = arith.op %v0, %v1, ... : T
with
%e0 = arith.expf %v0 : T to U
%e1 = arith.expf %v1 : T to U
...
%y.exp = arith.op %e0, %e1, ... : U
%y = arith.truncf %y.exp : U to T
This allows for "emulating" floating-point operations not supported on
a given target (such as bfloat operations or most arithmetic on 8-bit
floats) by extending those types to supported ones, performing the
arithmetic operation, and then truncating back to the original
type (which ensures appropriate rounding behavior).
The lowering of the extf and truncf ops introduced by this
transformation should be handled by subsequent passes.
Reviewed By: rsuderman
Differential Revision: https://reviews.llvm.org/D154539
This patch adds support for narrow bitwidth storage emulation. The goal is to support sub-byte type
codegen for LLVM CPU. Specifically, a type converter is added to convert memref of narrow bitwidth
(e.g., i4) into supported wider bitwidth (e.g., i8). Another focus of this patch is to populate the
pattern for int4 memref.load. memref.store pattern should be added in a seperate patch.
Reviewed By: hanchung, mravishankar
Differential Revision: https://reviews.llvm.org/D151519
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This patch updates all remaining uses of the deprecated functionality in
mlir/. This was done with clang-tidy as described below and further
modifications to GPUBase.td and OpenMPOpsInterfaces.td.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
```
Differential Revision: https://reviews.llvm.org/D151542
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Caveats include:
- This clang-tidy script probably has more problems.
- This only touches C++ code, so nothing that is being generated.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This first patch was created with the following steps. The intention is
to only do automated changes at first, so I waste less time if it's
reverted, and so the first mass change is more clear as an example to
other teams that will need to follow similar steps.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
4. Some changes have been deleted for the following reasons:
- Some files had a variable also named cast
- Some files had not included a header file that defines the cast
functions
- Some files are definitions of the classes that have the casting
methods, so the code still refers to the method instead of the
function without adding a prefix or removing the method declaration
at the same time.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
git restore mlir/lib/IR mlir/lib/Dialect/DLTI/DLTI.cpp\
mlir/lib/Dialect/Complex/IR/ComplexDialect.cpp\
mlir/lib/**/IR/\
mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp\
mlir/lib/Dialect/Vector/Transforms/LowerVectorMultiReduction.cpp\
mlir/test/lib/Dialect/Test/TestTypes.cpp\
mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp\
mlir/test/lib/Dialect/Test/TestAttributes.cpp\
mlir/unittests/TableGen/EnumsGenTest.cpp\
mlir/test/python/lib/PythonTestCAPI.cpp\
mlir/include/mlir/IR/
```
Differential Revision: https://reviews.llvm.org/D150123
These rely on the `ValueBounds` interace and its utility function to
compute constant bounds. This allows us to optimize `linalg.index`
values cast to integer types.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149538
This commutes the extension (`arith.extsi`, `arith.extui`) over the
following vector ops: `vector.broadcast`, `vector.shape_cast`,
`vector.transpose`, `vector.flat_transpose`.
I focused on these as I saw them getting created by vector unroll
patterns. Maybe except `vector.flat_transpose`.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149534
Allow to commute extension ops over `vector.insertelement` and
`vector.insert_strided_slice`.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149509
Add a new wraper type that represents either of `ExtSIOp` or `ExtUIOp`.
This is to simplify the code by using a single type, so that we do not
have to use templates or branching to handle both extension kinds.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149485
This enabled more optimization opportunities by moving
zero/sign-extension closer to the use.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149282
This moves zero/sign-extension ops closer to their use and exposes more
narrowing optimization opportunities.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D149233
This pass reduces the logical complexity of arith ops by choosing
narrowest supported operand bitwidth. On some targets like mobile GPUs,
narrower bitwidths also bring better runtime performance.
The first batch of rewrites handles a simple case of `arith.sitofp`
and `arith.uitofp` with zero/sign-extended inputs. In future revisions,
I plan to extend it with the following:
- Propagating sign/zero-extensions through bit-pattern-preserving ops,
e.g., vector transpose, broadcast, insertions/extractions.
- Handling `linalg.index` using the `ValueBounds` interface.
- Handling more arith ops.
Reviewed By: springerm, antiagainst
Differential Revision: https://reviews.llvm.org/D149118
Currently conversions to interfaces may happen implicitly (e.g.
`Attribute -> TypedAttr`), failing a runtime assert if the interface
isn't actually implemented. This change marks the `Interface(ValueT)`
constructor as explicit so that a cast is required.
Where it was straightforward to I adjusted code to not require casts,
otherwise I just made them explicit.
Depends on D148491, D148492
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D148493
ExpandOpsPass could only be configured via command line flags. Updated
to allowed constructing using the specified Options structure.
Reviewed By: NatashaKnk
Differential Revision: https://reviews.llvm.org/D148820
Add a new helper function for the type converter that takes care of
casting to the target type.
This is to avoid bugs where an incorrect cast function is used after
type conversion, e.g., `dyn_cast` or `cast`. These are not guaranteed to
work when type conversion fails, or when type conversion succeeds but
the provided type converted returned a type that a conversion pattern
did not expect.
I saw this being an issue in some SPIR-V passes and in mlir-hlo.
Exercise the new function in a couple of passes. As a side-effect, this
also made the code more concise.
Reviewed By: rriddle, mehdi_amini
Differential Revision: https://reviews.llvm.org/D148725
This revision adds an additional `reifyValueBounds` helper that reifies the IR with Arith ops instead of Affine ops. This is needed to support value bounds for integer types different from `index` in a subsequent revision.
Differential Revision: https://reviews.llvm.org/D146524
bf16 has a trivial truncation/extension behavior with F32 that
can be described in elementary arith operations. Include some
expansions to efficiently convert including rounding towards
infinity for f32 to bf16 truncation.
Reviewed By: jpienaar
Differential Revision: https://reviews.llvm.org/D147585
bf16 has a trivial truncation/extension behavior with F32 that
can be described in elementary arith operations. Include some
expansions to efficiently convert.
Reviewed By: jpienaar
Differential Revision: https://reviews.llvm.org/D147091
This patch extends the `createConst` method so that it can generate
constant vectors (it can already generate scalars). This change is
required to be able to apply the converter for `arith.floordivsi`
(i.e. `FloorDivSIOpConverter`) to vectors.
While `arith.floordivsi` is my main motivation for this change, this
patch should also allow other Arith ops to be converted in vector cases.
In my example, the Linalg vectorizer updates `arith.floordivsi` to
operate on vectors and hence the need for this change.
Differential Revision: https://reviews.llvm.org/D146741
This includes standard LIT tests and integration tests with the LLVM CPU
runner.
I plan to use this to implement `sitofp` in D146597.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D146606
`getAliasingOpOperands`/`getAliasingOpResults` now encodes OpOperand/OpResult, buffer relation and a degree of certainty. E.g.:
```
// aliasingOpOperands(%r) = {(%t, EQUIV, DEFINITE)}
// aliasingOpResults(%t) = {(%r, EQUIV, DEFINITE)}
%r = tensor.insert %f into %t[%idx] : tensor<?xf32>
// aliasingOpOperands(%r) = {(%t0, EQUIV, MAYBE), (%t1, EQUIV, MAYBE)}
// aliasingOpResults(%t0) = {(%r, EQUIV, MAYBE)}
// aliasingOpResults(%t1) = {(%r, EQUIV, MAYBE)}
%r = arith.select %c, %t0, %t1 : tensor<?xf32>
```
`BufferizableOpInterface::bufferRelation` is removed, as it is now part of `getAliasingOpOperands`/`getAliasingOpResults`.
This change allows for better analysis, in particular wrt. equivalence. This allows additional optimizations and better error checking (which is sometimes overly conservative). Examples:
* EmptyTensorElimination can eliminate `tensor.empty` inside `scf.if` blocks. This requires a modeling of equivalence: It is not a per-OpResult property anymore. Instead, it can be specified for each OpOperand and OpResult. This is important because `tensor.empty` may be eliminated only if all values on the SSA use-def chain to the final consumer (`tensor.insert_slice`) are equivalent.
* The detection of "returning allocs from a block" can be improved. (Addresses a TODO in `assertNoAllocsReturned`.) This allows us to bufferize IR such as "yielding a `tensor.extract_slice` result from an `scf.if` branch", which currently fails to bufferize because the alloc detection is too conservative.
* Better bufferization of loops. Aliases of the iter_arg can be yielded (even if they are not equivalent) without having to realloc and copy the entire buffer on each iteration.
The above-mentioned examples are not yet implemented with this change. This change just improves the BufferizableOpInterface, its implementations and related helper functions, so that better aliasing information is available for each op.
Differential Revision: https://reviews.llvm.org/D142129
* `getAliasingOpOperand` => `getAliasingOpOperands`
* `getAliasingOpResult` => `getAliasingOpResults`
Also a few minor code cleanups and better documentation.
Differential Revision: https://reviews.llvm.org/D142979
The previous name was incorrect. `None` does not mean that there is no buffer relation between two buffers (seems to imply that they do not alias for sure); instead it means that there is no further information available.
Differential Revision: https://reviews.llvm.org/D142870
Add a pass which do arith dialect ops optimization based on integer range analysis (only cmpi for now).
Differential Revision: https://reviews.llvm.org/D140629
As of several months ago, both ArithToLLVM and ArithToSPIRV have
native support for integer min and max operations. Since these are all
the targets available in MLIR core, the need to "expand" arith.minui,
arith.minsi, arith,maxsi, and arith.manxui to more primitive
operations is to longer present.
Therefore, the expanding of integer min and max operations in Arith,
while correct, is likely to lead to performance loss by way of
misoptimization further down the line, and is no longer needed for
anyone's correctness.
This change may break downstream tests, but will not affect the
semantics of MLIR programs.
arith.minf and arith.maxf have a lot of underlying complexity due to
the many different possible NaN and signed zero semantics available on
various platforms, and so removing their expansion is left to a future
commit.
Reviewed By: ThomasRaoux, Mogball
Differential Revision: https://reviews.llvm.org/D140856
Using `arith.mului_extended` makes it much simpler to emulate wide
integer multiplication.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D139776
The goal is to make the naming of the future `_extended` ops more
consistent. With unsigned addition, the carry value/flag and overflow
bit are the same, but this is not true when it comes to signed addition.
Also rename the second result from `carry` to `overflow`.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D139569
The pass was not checking for uninitialized states due to dead code.
This patch also makes LLVMFuncOp correctly return a null body when it is
external.
Fixes#58807
Depends on D139388
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D139389
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
MemRef has been accepting a general Attribute as memory space for
a long time. This commits updates bufferization side to catch up,
which allows downstream users to plugin customized symbolic memory
space. This also eliminates quite a few `getMemorySpaceAsInt`
calls, which is deprecated.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D138330
Add a new pass and conversions to emulate wide integer operations over memrefs.
The emulation is implemented on top of the existing pass to emulate wide integer arith ops.
Improve naming in the arith pass to avoid potential name clashes.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D135722
Print unsupported types on match failures.
Suggested by @Mogball and @jpienaar in D135204.
Reviewed By: Mogball
Differential Revision: https://reviews.llvm.org/D135673
This allows more precise control over which patterns to pick to
expand arithmetic ops. Previously ceil/floor division epxansion
is only available together with various min/max op expansion.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D135479
