Last patch required to avoid creating a temporary for the LHS when
dealing with `x([a,b]) = y`.
The code dealing with "ordered assignments" (where, forall, user and
vector subscripted assignments) is saving the evaluated RHS/LHS and
masks if they have write effects because this write effects should not
be evaluated when they affect entities that may be written to in other
contexts after the evaluation and before the re-evaluation.
But when dealing with write to storage allocated in the region for the
expression being evluated, there is no problem to re-evaluate the write:
it has no effect outside of the expression evaluation that owns the
allocation.
In the case of `x([a,b]) = y`, the temporary is created for the vector
subscript. Raising the HLFIR abstraction for simple array constructors
may be a good idea, but local temps are created in other contexts, so
this fix is more generic.
While experimenting with some more recent C++ features, I ran into
trouble with warnings from GCC 12.3.0 and 14.2.0. These warnings looked
legitimate, so I've tweaked the code to avoid them.
This PR intends to optimize away `hlfir.elemental` operations, which
leave temporary buffers (`allocmem`) in FIR. We typically see elemental
operations in the arguments of reduction intrinsics, so extending
`OptimizedBufferization` shall be the first solution to get heap-free
code.
Here we newly handle `minval`/`maxval` along with other reduction
intrinsics. Those functions over elemental become do loops. Furthermore,
we take the same action with `hlfir.designate` in order to inline more
intrinsics, which otherwise call runtime routines.
This patch adds more precise side effects to the current ops with memory
effects, allowing us to determine which OpOperand/OpResult/BlockArgument
the
operation reads or writes, rather than just recording the reading and
writing
of values. This allows for convenient use of precise side effects to
achieve
analysis and optimization.
Related discussions:
https://discourse.llvm.org/t/rfc-add-operandindex-to-sideeffect-instance/79243
The runtime API for copy-in copy-out currently only has an entry only
for the copy-out. This entry has a "skipInit" boolean that is never set
to false by lowering and it does not deal with the deallocation of the
temporary.
The generated code was a mix of inline code and runtime calls This is not a big deal,
but this is unneeded compiler and generated code complexity.
With assumed-rank, it is also more cumbersome to establish a
temporary descriptor.
Instead, this patch:
- Adds a CopyInAssignment API that deals with establishing the temporary
descriptor and does the copy.
- Removes unused arg to CopyOutAssign, and pushes
destruction/deallocation responsibility inside it.
Note that this runtime API are still not responsible for deciding the
need of copying-in and out. This is kept as a separate runtime call to
IsContiguous, which is easier to inline/replace by inline code with the
hope of removing the copy-in/out calls after user function inlining.
@vzakhari has already shown that always inlining all the copy part
increase Fortran compilation time due to loop optimization attempts for
loops that are known to have little optimization profitability (the
variable being copied from and to is not contiguous).
This is a heavy process, and it can trigger a massive explosion in
adding block arguments. While potentially reducing the code size, the
resulting merged blocks with arguments are hiding some of the def-use
chain and can even hinder some further analyses/optimizations: a merge
block does not have it's own path-sensitive context, instead the context
is merged from all the predecessors.
Previous behavior can be restored by passing:
{test-convergence region-simplify=aggressive}
to the canonicalize pass.
In some cases where we have an `hlfir.no_reassoc` operation, the
bufferization pass could not earse the hlfir.destroy op during the
`hlfir.associate` op conversion as show in the example below.
```
func.func @double_free(%arg0: !fir.boxchar<1>) {
%c5 = arith.constant 5 : index
%true = arith.constant true
%0 = hlfir.as_expr %arg0 move %true : (!fir.boxchar<1>, i1) -> !hlfir.expr<!fir.char<1,?>>
%1 = hlfir.no_reassoc %0 : !hlfir.expr<!fir.char<1,?>>
%2:3 = hlfir.associate %1 typeparams %c5 {adapt.valuebyref} : (!hlfir.expr<!fir.char<1,?>>, index) -> (!fir.boxchar<1>, !fir.ref<!fir.char<1,?>>, i1)
fir.call @noop(%2#0) : (!fir.boxchar<1>) -> ()
hlfir.end_associate %2#1, %2#2 : !fir.ref<!fir.char<1,?>>, i1
hlfir.destroy %0 : !hlfir.expr<!fir.char<1,?>>
return
}
func.func private @noop(!fir.boxchar<1>)
```
The bufferization pass is looking at uses of its source `%1` that is the
result of an `hlfir.no_reassoc` operation. In order to avoid double free
generation, also look at the indirection in presence of
`hlfir.no_reassoc`.
hlfir.declare is in charge of ensuring that the lower bounds of its
"hlfir entity" output are the ones of the source program. For
non-allocatable/non-pointer assumed-ranks where the input descriptor
lower bounds may not be ones, the hlfir.declare needs to be lowered to
an hlfir.rebox_assumed_rank to set the lower bounds to ones.
The pass constructor can be generated automatically.
This pass is module-level and then runs on all relevant intrinsic
operations inside of the module, no matter what top level operation they
are inside of.
The pass constructor can be generated automatically.
This pass is module-level and then runs on all of the relevant HLFIR
operations inside of the module, no matter what top level operation they
are inside of.
The pass constructor can be generated automatically.
This pass is module-level and then runs on all relevant intrinsic
operations inside of the module, no matter what top level operation they
are inside of.
This pass is module-level and then runs on all operations implementing
the ordered assignment interface. It should not matter which top-level
operation the assignment is inside of.
This commit removes some unnecessary boilerplate that we can generate
automatically with tablegen.
This means that this pass will also run on hlfir elemental operations
which are not inside of functions.
See RFC:
https://discourse.llvm.org/t/rfc-add-an-interface-for-top-level-container-operations
Some of the changes are from moving the declaration and definition of
the constructor into tablegen (as requested during code review of
another pass).
This means that this pass will also run on hlfir elemental operations
which are not inside of functions.
See RFC:
https://discourse.llvm.org/t/rfc-add-an-interface-for-top-level-container-operations
Some of the changes are from moving the declaration and definition of
the constructor into tablegen (as requested during code review of
another pass).
While I was updating the tests I noticed that the optimized
bufferization pass and some cse were missing from the optimized pipeline
in flang/test/Driver/mlir-pass-pipeline.f90. I fixed this in this
commit.
This means that this pass will also run on hlfir intrinsics which are
not inside of functions.
See RFC:
https://discourse.llvm.org/t/rfc-add-an-interface-for-top-level-container-operations
Some of the changes are from moving the declaration and definition of
the constructor into tablegen (as requested during code review of
another pass).
The number of operations dedicated to CUF grew and where all still in
FIR. In order to have a better organization, the CUF operations,
attributes and code is moved into their specific dialect and files. CUF
dialect is tightly coupled with HLFIR/FIR and their types.
The CUF attributes are bundled into their own library since some
HLFIR/FIR operations depend on them and the CUF dialect depends on the
FIR types. Without having the attributes into a separate library there
would be a dependency cycle.
The HLFIR pass lowering WHERE (hlfir.where op) was too aggressive in its
hoisting of scalar sub-expressions from LHS/RHS/MASKS outside of the
loops generated for the WHERE construct.
This violated F'2023 10.2.3.2 point 10 that stipulated that elemental
operations must be evaluated only for elements corresponding to true
values, because scalar operations are still elemental, and hoisting them
is invalid if they could have side effects (e.g, division by zero) and
if the MASK is always false (i.e., the loop body is never evaluated).
The difficulty is that 10.2.3.2 point 9 mandates that nonelemental
function must be evaluated before the loops. So it is not possible to
simply stop hoisting non hlfir.elemental operations.
Marking calls with an elemental/nonelemental attribute would not allow
the pass to be correct if inlining is run before and drops this
information, beside, extracting the argument tree that may have been
CSE-ed with the rest of the expression evaluation would be a bit
combursome.
Instead, lower nonelemental calls into a new hlfir.exactly_once
operation that will allow retaining the information that the operations
contained inside its region must be hoisted. This allows inlining to
operate before if desired in order to improve alias analysis.
The LowerHLFIROrderedAssignments pass is updated to only hoist the
operations contained inside hlfir.exactly_once bodies.
The lowering produces fir.dummy_scope operation if the current
function has dummy arguments. Each hlfir.declare generated
for a dummy argument is then using the result of fir.dummy_scope
as its dummy_scope operand. This is only done for HLFIR.
I was not able to find a reliable way to identify dummy symbols
in `genDeclareSymbol`, so I added a set of registered dummy symbols
that is alive during the variables instantiation for the current
function. The set is initialized during the mapping of the dummy
argument symbols to their MLIR values. It is reset right after
all variables are instantiated - this is done to avoid generating
hlfir.declare operations with dummy_scope for the clones of
the dummy symbols (e.g. this happens with OpenMP privatization).
If this can be done in a cleaner way, please advise.
The new operation is just an abstract attribute that is attached to
[hl]fir.declare operations of dummy arguments of a subroutine.
Dummy arguments of the same subroutine refer to the same
fir.dummy_scope, so they can be recognized as such during FIR AliasAnalysis.
Note that the fir.dummy_scope must be specific to the runtime
instantiation of a subroutine, so any MLIR inlining/cloning should duplicate and
unique it vs using the same fir.dummy_scope for different runtime instantiations.
This is why I made it an operation rather than an attribute.
The new operation uses a write effect on DebuggingResource, same as
[hl]fir.declare, to avoid optimizing it away.
…ted. (#89998)" (#90250)
This partially reverts commit 7aedd7dc75.
This change removes calls to the deprecated member functions. It does
not mark the functions deprecated yet and does not disable the
deprecation warning in TypeSwitch. This seems to cause problems with
MSVC.
In #83253 @matthias-springer pointed out that LowerHLFIRIntrinsics.cpp
should not be using rewrite patterns with the dialect conversion driver.
The intention of this pass is to lower HLFIR intrinsic operations into
FIR so it conceptually fits dialect conversion. However, dialect
conversion is much stricter about changing types when replacing
operations. This pass sometimes looses track of array bounds, resulting
in replacements with operations with different but compatible types
(expressions of the same rank and element types but with or without
compile time known array bounds). This is difficult to accommodate with
the dialect conversion driver and so I have changed to use the greedy
pattern rewriter.
There is a lot of test churn because the greedy pattern rewriter also
performs canonicalization.
This fix is a temporary workaround. `LowerHLFIRIntrinsics.cpp` should be
using the greedy pattern rewriter or a manual IR traversal. All patterns
in this file are rewrite patterns. The test failure was caused by
`replaceAllUsesWith`, which is not supported by the dialect conversion;
additional asserts were added recently to prevent incorrect API usage.
These trigger now.
Alternatively, turning the patterns into conversion patterns and
specifying a type converter may work.
Failing test case:
`Fortran/gfortran/regression/gfortran-regression-compile-regression__inline_matmul_14_f90.test`
In certain case "extreme" values like Nan, Inf and 0xffffffff could lead
to generating different code via the inline-generated intrinsics vs the
versions in the runtimes (and other compilers like gfortran). There are
some examples I was using for testing in
https://godbolt.org/z/x4EfqEss5.
This changes the generation for the intrinsics to be more like the
runtimes, using a condition that is similar to:
isFirst || (prev != prev && elem == elem) || elem < prev
The middle part is only used for floating point operations, and checks
if the values are Nan. This should then hopefully make the logic closer
to - return the first element with the lowest value, with Nans ignored
unless there are only Nans. The initial limit value for floats are also
changed from the largest float to Inf, to make sure it is handled
correctly.
The integer reductions are also changed to use a similar scheme to make
sure they work with masked values. This means that the preamble after
the loop can be removed.
Context: Conversion patterns provide a `ConversionPatternRewriter` to
modify the IR. `ConversionPatternRewriter` provides the public API. Most
function calls are forwarded/handled by `ConversionPatternRewriterImpl`.
The dialect conversion uses the listener infrastructure to get notified
about op/block insertions.
In the current design, `ConversionPatternRewriter` inherits from both
`PatternRewriter` and `Listener`. The conversion rewriter registers
itself as a listener. This is problematic because listener functions
such as `notifyOperationInserted` are now part of the public API and can
be called from conversion patterns; that would bring the dialect
conversion into an inconsistent state.
With this commit, `ConversionPatternRewriter` no longer inherits from
`Listener`. Instead `ConversionPatternRewriterImpl` inherits from
`Listener`. This removes the problematic public API and also simplifies
the code a bit: block/op insertion notifications were previously
forwarded to the `ConversionPatternRewriterImpl`. This is no longer
needed.
This is one option for attempting to move genMinMaxlocReductionLoop to a
better location. It moves it into Transforms and makes HLFIRTranforms
depend upon FIRTransforms.
It passes a build locally, both with and without -DBUILD_SHARED_LIBS,
and does OK on the windows CI.
I was looking through to check whether Nan was being handled correctly,
and couldn't work out why simple cases were behaving differently than
they should. It turns out the initial limit values was backwards for
minloc/maxloc reductions in general. This fixes that, introduced in
#79469.
The newly introduced `CUDAAttribute` is meant for CUDA attributes
associated with variable. In order to not clash with the future
attribute for function/subroutine, rename `CUDAAttribute` to
`CUDADataAttribute`.
This is a first simple patch to introduce a new FIR attribute to carry
the CUDA variable attribute information to hlfir.declare and fir.declare
operations. It currently lowers this information for local variables.
The texture attribute is omitted since it is rejected by semantic and
will not make its way to MLIR.
This new attribute is added as optional attribute to the hlfir.declare
and fir.declare operations.
https://github.com/llvm/llvm-project/pull/80683 revealed that
hlfir.as_expr was propagating the temporary buffer for polymorphic
values as an allocatable while codegen later expects to be working with
fir.box/fir.class but not fir.ref<box/class> when processing the
operations using the hlfir.as_expr result.
Dereference the temporary allocatable as soon as it is created.
The MSCV build doesn't allow the constexpr isMax variable to be used in lambda
without a capture. The -Weverything build does not allow isMax to be used in a
lambda capture as it is a constexpr. I've removed the constexpr as it shouldn't
be necessary.
This change makes the callback consistent with
`notifyOperationInserted`: both now notify about IR insertion, not IR
creation. See also #78988.
This change also simplifies the dialect conversion: it is no longer
necessary to override the `inlineRegionBefore` method. All information
that is necessary for rollback is provided with the
`notifyBlockInserted` callback.
Currently the lowering of a minloc intrinsic with a mask will look something
like:
%e = hlfir.elemental %shape ({
...
})
%m = hlfir.minloc %array mask %e
hlfir.assign %m to %result
hlfir.destroy %m
The elemental will be expanded into a temporary+loop, the minloc into a
FortranAMinloc call (which hopefully gets simplified to a specialized call that
can be inlined at the call site), and the assign might get expanded to a
FortranAAssign. It would be better to generate the entire construct as single
loop if we can - one that performs the minloc calculation with the mask
elemental computed inline.
This patch attempt to do that, adding a hlfir version of the expansion code
from SimplifyIntrinsics that turns an minloc+elemental into a single combined
loop nest. It attempts to reuse the methods in genMinlocReductionLoop for
constructing the loop with a modified loop body. The declaration for the
function is currently in Optimizer/Support/Utils.h, but there might be a better
place for it.
It is added as part of the OptimizedBufferizationPass, like the
similar count/any/all that have been added recently.
The pattern rewriter documentation states that "*all* IR mutations [...]
are required to be performed via the `PatternRewriter`." This commit
adds two functions that were missing from the rewriter API:
`moveOpBefore` and `moveOpAfter`.
After an operation was moved, the `notifyOperationInserted` callback is
triggered. This allows listeners such as the greedy pattern rewrite
driver to react to IR changes.
This commit narrows the discrepancy between the kind of IR modification
that can be performed and the kind of IR modifications that can be
listened to.
When running the `flang/test/HLFIR/simplify-hlfir-intrinsics.fir` test
case on AIX we encounter issues building op as they are not found in the
mlir context:
```
LLVM ERROR: Building op `arith.subi` but it isn't known in this MLIRContext: the dialect may not be loaded or this operation hasn't been added by the dialect. See also https://mlir.llvm.org/getting_started/Faq/#registered-loaded-dependent-whats-up-with-dialects-management
LLVM ERROR: Building op `hlfir.yield_element` but it isn't known in this MLIRContext: the dialect may not be loaded or this operation hasn't been added by the dialect. See also https://mlir.llvm.org/getting_started/Faq/#registered-loaded-dependent-whats-up-with-dialects-management
LLVM ERROR: Building op `hlfir.yield_element` but it isn't known in this MLIRContext: the dialect may not be loaded or this operation hasn't been added by the dialect. See also https://mlir.llvm.org/getting_started/Faq/#registered-loaded-dependent-whats-up-with-dialects-management
```
The issue is caused by the "Merge disjoint stack slots" pass and the
error is not present if the source is built with `-mllvm
--no-stack-coloring`
Thanks to investigation by @stefanp-ibm we found that "the
initializer_list {inputIndices[1], inputIndices[0]} has a lifetime that
only exists for the range of the constructor for ValueRange. Once we get
to stack coloring we merge the stack slot for that element with another
stack slot and then it gets overwritten which corrupts
transposedIndices"
The changes below prevents the corruption of transposedIndices and
passes the test case.
Co-authored-by: Mark Danial <mark.danial@ibm.com>
This commit renames 4 pattern rewriter API functions:
* `updateRootInPlace` -> `modifyOpInPlace`
* `startRootUpdate` -> `startOpModification`
* `finalizeRootUpdate` -> `finalizeOpModification`
* `cancelRootUpdate` -> `cancelOpModification`
The term "root" is a misnomer. The root is the op that a rewrite pattern
matches against
(https://mlir.llvm.org/docs/PatternRewriter/#root-operation-name-optional).
A rewriter must be notified of all in-place op modifications, not just
in-place modifications of the root
(https://mlir.llvm.org/docs/PatternRewriter/#pattern-rewriter). The old
function names were confusing and have contributed to various broken
rewrite patterns.
Note: The new function names use the term "modify" instead of "update"
for consistency with the `RewriterBase::Listener` terminology
(`notifyOperationModified`).
When dealing with overlaps in user defined assignments, some entities
with descriptors (fir.box) may be saved without descriptors. The current
code was replacing the original box entity with the "raw" copy with a
simple cast instead of creating a box for the copy. This patch ensures a
fir.embox is emitted instead.
Certain compilers do not seem to like the static assert with a string, causing
a implicit conversion. It can be removed as it should not be reachable and the
mlir::failure should handle it correctly in case it is.
This adds a ReductionElementalConversion transform to
OptimizedBufferizationPass, taking hlfir::count(hlfir::elemental) and
generating the inline loop to perform the count of true elements. This
lets us generate a single loop instead of ending up as two plus a
temporary.
Any and All should be able to share the same code with a different
function/initial value.
The adds a hlfir minloc intrinsic, similar to the minval intrinsic
already added, to help in the lowering of minloc. The idea is to later
add maxloc too, and from there add a simplification for producing minloc
with inlined elemental and hopefully less temporaries.
This got "lost" in the HLFIR transformation. This patch applies the old
attribute to the AssociateOp that needs it, and forwards it to the
AllocaOp that is generated when lowering to FIR.
To avoid the overhead of deallocating allocatable components of the
elemental temporary result on every iteration of the elemental operation,
we can use a shallow copy instead of deep-copy assign.
