This revision extends the GPU dialect with ops that can be lowered to
host-oriented sparse matrix library calls (in this case cuSparse focused
although the ops could be generalized to support more GPUs in principle).
This will allow the "sparse compiler pipeline" to accelerate sparse operations
(see follow up revisions with examples of this).
For some background;
https://discourse.llvm.org/t/sparse-compiler-and-gpu-code-generation/69786/2
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D150152
The MLIR classes Type/Attribute/Operation/Op/Value support
cast/dyn_cast/isa/dyn_cast_or_null functionality through llvm's doCast
functionality in addition to defining methods with the same name.
This change begins the migration of uses of the method to the
corresponding function call as has been decided as more consistent.
Note that there still exist classes that only define methods directly,
such as AffineExpr, and this does not include work currently to support
a functional cast/isa call.
Caveats include:
- This clang-tidy script probably has more problems.
- This only touches C++ code, so nothing that is being generated.
Context:
- https://mlir.llvm.org/deprecation/ at "Use the free function variants
for dyn_cast/cast/isa/…"
- Original discussion at https://discourse.llvm.org/t/preferred-casting-style-going-forward/68443
Implementation:
This first patch was created with the following steps. The intention is
to only do automated changes at first, so I waste less time if it's
reverted, and so the first mass change is more clear as an example to
other teams that will need to follow similar steps.
Steps are described per line, as comments are removed by git:
0. Retrieve the change from the following to build clang-tidy with an
additional check:
https://github.com/llvm/llvm-project/compare/main...tpopp:llvm-project:tidy-cast-check
1. Build clang-tidy
2. Run clang-tidy over your entire codebase while disabling all checks
and enabling the one relevant one. Run on all header files also.
3. Delete .inc files that were also modified, so the next build rebuilds
them to a pure state.
4. Some changes have been deleted for the following reasons:
- Some files had a variable also named cast
- Some files had not included a header file that defines the cast
functions
- Some files are definitions of the classes that have the casting
methods, so the code still refers to the method instead of the
function without adding a prefix or removing the method declaration
at the same time.
```
ninja -C $BUILD_DIR clang-tidy
run-clang-tidy -clang-tidy-binary=$BUILD_DIR/bin/clang-tidy -checks='-*,misc-cast-functions'\
-header-filter=mlir/ mlir/* -fix
rm -rf $BUILD_DIR/tools/mlir/**/*.inc
git restore mlir/lib/IR mlir/lib/Dialect/DLTI/DLTI.cpp\
mlir/lib/Dialect/Complex/IR/ComplexDialect.cpp\
mlir/lib/**/IR/\
mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp\
mlir/lib/Dialect/Vector/Transforms/LowerVectorMultiReduction.cpp\
mlir/test/lib/Dialect/Test/TestTypes.cpp\
mlir/test/lib/Dialect/Transform/TestTransformDialectExtension.cpp\
mlir/test/lib/Dialect/Test/TestAttributes.cpp\
mlir/unittests/TableGen/EnumsGenTest.cpp\
mlir/test/python/lib/PythonTestCAPI.cpp\
mlir/include/mlir/IR/
```
Differential Revision: https://reviews.llvm.org/D150123
This patch is primarily about the change in
"mlir/tools/mlir-cpu-runner/CMakeLists.txt". LLJIT needs access to
symbols (e.g. llvm_orc_registerEHFrameSectionWrapper) that will be
defined in the executable when LLVM is linked statically. This change is
consistent with how other tools within LLVM use LLJIT. It is required to
make sure that:
```
$ mlir-cpu-runner --host-supports-jit
```
correctly returns `true` on platforms that do support JITting (in my
case that's AArch64 Linux).
The change in "mlir/lib/ExecutionEngine/CMakeLists.txt" is required to
avoid ODR violations when symbols from `mlir-cpu-runner` are exported
and when loading `libmlir_async_runtime.so` in `mlir-cpu-runner`.
Specifically, to avoid `EnableABIBreakingChecks` being defined twice.
For more context:
* https://github.com/llvm/llvm-project/issues/61712
* https://github.com/llvm/llvm-project/issues/61856
* https://reviews.llvm.org/D146935 (this PR)
This change relands ccdcfad081Fixes#61856
Differential Revision: https://reviews.llvm.org/D146935
Without explicitly unregistering you will get
```
'cuMemHostRegister(ptr, sizeBytes, 0)' failed with 'CUDA_ERROR_HOST_MEMORY_ALREADY_REGISTERED'
```
in CUDA (for example) after repeated runs (e.g., during benchmarking the same kernel).
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D147277
This patch adds support for `-mattr` and `-march` in mlir-cpu-runner.
With this change, one should be able to consistently use mlir-cpu-runner
for MLIR's integration tests (instead of e.g. resorting to lli when some
additional flags are needed). This is demonstrated in
concatenate_dim_1.mlir.
In order to support the new flags, this patch makes sure that
MLIR's ExecutionEngine/JITRunner (that mlir-cpu-runner is built on top of):
* takes into account the new command line flags when creating
TargetMachine,
* avoids recreating TargetMachine if one is already available,
* creates LLVM's DataLayout based on the previously configured
TargetMachine.
This is necessary in order to make sure that the command line
configuration is propagated correctly to the backend code generator.
A few additional updates are made in order to facilitate this change,
including support for debug dumps from JITRunner.
Differential Revision: https://reviews.llvm.org/D146917
This patch supports the processing of dialect attributes attached to top-level
module-type operations during MLIR-to-LLVMIR lowering.
This approach modifies the `mlir::translateModuleToLLVMIR()` function to call
`ModuleTranslation::convertOperation()` on the top-level operation, after its
body has been lowered. This, in turn, will get the
`LLVMTranslationDialectInterface` object associated to that operation's dialect
before trying to use it for lowering prior to processing dialect attributes
attached to the operation.
Since there are no `LLVMTranslationDialectInterface`s for the builtin and GPU
dialects, which define their own module-type operations, this patch also adds
and registers them. The requirement for always calling
`mlir::registerBuiltinDialectTranslation()` before any translation of MLIR to
LLVM IR where builtin module operations are present is introduced. The purpose
of these new translation interfaces is to succeed when processing module-type
operations, allowing the lowering process to continue and to prevent the
introduction of failures related to not finding such interfaces.
Differential Revision: https://reviews.llvm.org/D145932
The default and pre-link pipeline builders currently require you to
call a separate method for optimization level O0, even though they
have perfectly well-defined O0 optimization pipelines.
Accept O0 optimization level and call buildO0DefaultPipeline()
internally, so all consumers don't need to repeat this.
Differential Revision: https://reviews.llvm.org/D146200
Replace references to enumerate results with either result_pairs
(reference wrapper type) or structured bindings. I did not use
structured bindings everywhere as it wasn't clear to me it would
improve readability.
This is in preparation to the switch to zip semantics which won't
support non-const lvalue reference to elements:
https://reviews.llvm.org/D144503.
I chose to use values instead of const lvalue-refs because MLIR is
biased towards avoiding `const` local variables. This won't degrade
performance because currently `result_pair` is cheap to copy (size_t
+ iterator), and in the future, the enumerator iterator dereference
will return temporaries anyway.
Reviewed By: dblaikie
Differential Revision: https://reviews.llvm.org/D146006
The old "pointer/index" names often cause confusion since these names clash with names of unrelated things in MLIR; so this change rectifies this by changing everything to use "position/coordinate" terminology instead.
In addition to the basic terminology, there have also been various conventions for making certain distinctions like: (1) the overall storage for coordinates in the sparse-tensor, vs the particular collection of coordinates of a given element; and (2) particular coordinates given as a `Value` or `TypedValue<MemRefType>`, vs particular coordinates given as `ValueRange` or similar. I have striven to maintain these distinctions
as follows:
* "p/c" are used for individual position/coordinate values, when there is no risk of confusion. (Just like we use "d/l" to abbreviate "dim/lvl".)
* "pos/crd" are used for individual position/coordinate values, when a longer name is helpful to avoid ambiguity or to form compound names (e.g., "parentPos"). (Just like we use "dim/lvl" when we need a longer form of "d/l".)
I have also used these forms for a handful of compound names where the old name had been using a three-letter form previously, even though a longer form would be more appropriate. I've avoided renaming these to use a longer form purely for expediency sake, since changing them would require a cascade of other renamings. They should be updated to follow the new naming scheme, but that can be done in future patches.
* "coords" is used for the complete collection of crd values associated with a single element. In the runtime library this includes both `std::vector` and raw pointer representations. In the compiler, this is used specifically for buffer variables with C++ type `Value`, `TypedValue<MemRefType>`, etc.
The bare form "coords" is discouraged, since it fails to make the dim/lvl distinction; so the compound names "dimCoords/lvlCoords" should be used instead. (Though there may exist a rare few cases where is is appropriate to be intentionally ambiguous about what coordinate-space the coords live in; in which case the bare "coords" is appropriate.)
There is seldom the need for the pos variant of this notion. In most circumstances we use the term "cursor", since the same buffer is reused for a 'moving' pos-collection.
* "dcvs/lcvs" is used in the compiler as the `ValueRange` analogue of "dimCoords/lvlCoords". (The "vs" stands for "`Value`s".) I haven't found the need for it, but "pvs" would be the obvious name for a pos-`ValueRange`.
The old "ind"-vs-"ivs" naming scheme does not seem to have been sustained in more recent code, which instead prefers other mnemonics (e.g., adding "Buf" to the end of the names for `TypeValue<MemRefType>`). I have cleaned up a lot of these to follow the "coords"-vs-"cvs" naming scheme, though haven't done an exhaustive cleanup.
* "positions/coordinates" are used for larger collections of pos/crd values; in particular, these are used when referring to the complete sparse-tensor storage components.
I also prefer to use these unabbreviated names in the documentation, unless there is some specific reason why using the abbreviated forms helps resolve ambiguity.
In addition to making this terminology change, this change also does some cleanup along the way:
* correcting the dim/lvl terminology in certain places.
* adding `const` when it requires no other code changes.
* miscellaneous cleanup that was entailed in order to make the proper distinctions. Most of these are in CodegenUtils.{h,cpp}
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D144773
Love or hate it, but the vector.print operation was the very
first operation that actually made "end-to-end" CHECK integration
testing possible for MLIR. This revision adds support for
the -until recently- less common but important floating-point
types f16 and bf16.
This will become useful for accelerator specific testing (e.g. NVidia GPUs)
Reviewed By: wrengr
Differential Revision: https://reviews.llvm.org/D145207
This is to prepare for implementing the C API for reading a COO tensor to the
given buffers for indices and values.
Reviewed By: wrengr
Differential Revision: https://reviews.llvm.org/D143861
When `LLVM_ENABLE_PIC = OFF`, shared libraries cannot be built against code
that's compiled without -fPIC. Example error message:
``
ld.lld: error: relocation R_X86_64_32 cannot be used against local symbol;
recompile with -fPIC
>>> defined in lib/libLLVMSupport.a(StringMap.cpp.o)
>>> referenced by StringMap.cpp
>>> StringMap.cpp.o:(llvm::StringMapImpl::StringMapImpl(unsigned
>>> int, unsigned int)) in archive lib/libLLVMSupport.a
``
Similar to [how libclang handles
this](https://github.com/llvm/llvm-project/blob/main/clang/tools/clang-shlib/CMakeLists.txt#L2-L4),
skip building these shared libraries when `LLVM_ENABLE_PIC = OFF`.
Differential Revision: https://reviews.llvm.org/D142941
The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.
The HIP CMake files expect to find their own dependencies and don't
use or respect PATHS or HINTS, relying on CMAKE_PREFIX_PATH to contain
/opt/rocm and /opt/rocm/hip . This is not great for the rest of the
build. Therefore, copy the CMake prefix path, add the ROCm
directories, find HIP, and reset the path to its old value.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D142391
This fixes a warning on MacOS:
warning: 'sprintf' is deprecated: This function is provided for compatibility
reasons only. Due to security concerns inherent in the design of sprintf(3),
it is highly recommended that you use snprintf(3) instead.
Replace void* with std::byte* in AsyncRuntime to make it clear that these pointers point to a memory region.
Reviewed By: ezhulenev
Differential Revision: https://reviews.llvm.org/D140428
This is a fairly large changeset, but it can be broken into a few
pieces:
- `llvm/Support/*TargetParser*` are all moved from the LLVM Support
component into a new LLVM Component called "TargetParser". This
potentially enables using tablegen to maintain this information, as
is shown in https://reviews.llvm.org/D137517. This cannot currently
be done, as llvm-tblgen relies on LLVM's Support component.
- This also moves two files from Support which use and depend on
information in the TargetParser:
- `llvm/Support/Host.{h,cpp}` which contains functions for inspecting
the current Host machine for info about it, primarily to support
getting the host triple, but also for `-mcpu=native` support in e.g.
Clang. This is fairly tightly intertwined with the information in
`X86TargetParser.h`, so keeping them in the same component makes
sense.
- `llvm/ADT/Triple.h` and `llvm/Support/Triple.cpp`, which contains
the target triple parser and representation. This is very intertwined
with the Arm target parser, because the arm architecture version
appears in canonical triples on arm platforms.
- I moved the relevant unittests to their own directory.
And so, we end up with a single component that has all the information
about the following, which to me seems like a unified component:
- Triples that LLVM Knows about
- Architecture names and CPUs that LLVM knows about
- CPU detection logic for LLVM
Given this, I have also moved `RISCVISAInfo.h` into this component, as
it seems to me to be part of that same set of functionality.
If you get link errors in your components after this patch, you likely
need to add TargetParser into LLVM_LINK_COMPONENTS in CMake.
Differential Revision: https://reviews.llvm.org/D137838
Add printMemref for complex data types and index type. Add printMemref for 1d
type beyond f32.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D139475
The bytecode reader currently has no mechanism that allows for directly referencing
data from the input buffer safely. This commit adds shared_ptr<SourceMgr> overloads
that provide an explicit and safe way of extending the lifetime of the input. The usage of
these new overloads is adopted in all of our tooling, and is implicitly used in the filename
only parser methods.
Differential Revision: https://reviews.llvm.org/D139366
This change cleans up the conversion pass re the "dim"-vs-"lvl" and "sizes"-vs-"shape" distinctions of the runtime. A quick synopsis includes:
* Adds new `SparseTensorStorageBase::getDimSize` method, with `sparseDimSize` wrapper in SparseTensorRuntime.h, and `genDimSizeCall` generator in SparseTensorConversion.cpp
* Changes `genLvlSizeCall` to perform no logic, just generate the function call.
* Adds `createOrFold{Dim,Lvl}Call` functions to handle the logic of replacing `gen{Dim,Lvl}SizeCall` with constants whenever possible. The `createOrFoldDimCall` function replaces the old `sizeFromPtrAtDim`.
* Adds `{get,fill}DimSizes` functions for iterating `createOrFoldDimCall` across the whole type. These functions replace the old `sizesFromPtr`.
* Adds `{get,fill}DimShape` functions for lowering a `ShapedType` into constants. These functions replace the old `sizesFromType`.
* Changes the `DimOp` rewrite to do the right thing.
* Changes the `ExpandOp` rewrite to compute the proper expansion size.
Depends On D138365
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D139165
This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
This commit updates how the `SparseTensorConversion` pass handles `NewOp`. It breaks up the underlying `openSparseTensor` function into two parts (`SparseTensorReader::create` and `SparseTensorReader::readSparseTensor`) so that the pass can inject code for constructing `lvlSizes` between those two parts. Migrating the construction of `lvlSizes` out of the runtime and into the pass is a necessary first step toward fully supporting non-permutations. (The alternative would be for the pass to generate a `FuncOp` for performing the construction and then passing that to the runtime; which doesn't seem to have any benefits over the design of this commit.) And since the pass now generates the code to call these two functions, this change also removes the `Action::kFromFile` value from the enum used by `_mlir_ciface_newSparseTensor`.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D138363
In particular, this silences warnings from [-Wsign-compare].
This is a revised version of D137735, which got reverted due to a sign-comparison warning on LLVM's Windows buildbot (which was not on MLIR's Windows buildbot). Differences vs the previous differential:
* `vectorToMemref` now uses `detail::checkOverflowCast` to silence the warning that caused the the previous differential to get reverted.
* `MEMREF_GET_USIZE` now uses `detail::checkOverflowCast` rather than `static_cast`
* `ASSERT_USIZE_EQ` added to abbreviate another common idiom, and to ensure that we use `detail::safelyEQ` everywhere (to silence a few other warnings)
* A couple for-loops now use `index_type` for the induction variable, since their upper bound uses that typedef too. (Namely `_mlir_ciface_getSparseTensorReaderDimSizes` and `_mlir_ciface_outSparseTensorWriterNext`)
Depends on D138149
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D137998
This is a reposting of D137737, which got reverted when D137735 did. There are no changes other than rebasing.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D138000
This patch fixes:
mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp:195:30: warning:
cast from type ‘const long unsigned int*’ to type ‘void*’ casts away
qualifiers [-Wcast-qual]
This patch fixes:
mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp:296:31: error:
comparison of integers of different signs: 'int64_t' (aka 'long')
and 'const uint64_t' (aka 'const unsigned long')
[-Werror,-Wsign-compare]
mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp:297:67: error:
comparison of integers of different signs: 'int64_t' (aka 'long')
and 'const uint64_t' (aka 'const unsigned long')
[-Werror,-Wsign-compare]
mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp:298:31: error:
comparison of integers of different signs: 'int64_t' (aka 'long') and
'const uint64_t' (aka 'const unsigned long') [-Werror,-Wsign-compare]
mlir/lib/ExecutionEngine/SparseTensorRuntime.cpp:479:30: error:
comparison of integers of different signs: 'int64_t' (aka 'long')
and 'const uint64_t' (aka 'const unsigned long')
[-Werror,-Wsign-compare]
In particular, this silences warnings from [-Wsign-compare].
Depends On D137681
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D137735
Systematically updates the SparseTensorRuntime to properly distinguish tensor-dimensions from storage-levels (and their associated ranks, shapes, sizes, indices, etc). With a few exceptions which are noted in the code, this ensures the runtime has all the **semantic** changes necessary to support non-permutations.
(Whereas **operationally**, since we're still using `std::vector<uing64_t>` to represent the mappings, there's no way to pass in any interesting non-permutations. Changing the representation to `std::function` will be done in a separate differential.)
Depends On D137680
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D137681
This allows the MLIR transformer to see the command line options and
make desicions based on them. No change upstream, but my use-case is to
look at the entry point name and type to make sure I can use them.
Differential Revision: https://reviews.llvm.org/D137861
This change make sure that ExecutionEngine's pass pipeline is identical to one
used by clang. Previously, SLPVectorization was not enabled which caused
differences in code...
...generation.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D137248
Move the SparseTensorEnums library out of the ExecutionEngine directory and into Dialect/SparseTensor/IR.
Depends On D136002
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D136005
This differential splits the SparseTensorEnums library out from the SparseTensorRuntime library. The actual moving of files will be handled in the next differential.
Depends On D135996
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D136002
This change is to make way for reusing the DimLevelType enum in lieu of the SparseTensorEncodingAttr::DimLevelType enum, but broken out to make it quick and easy to review
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D135995
