9e739fdb85
D154280 (landed in64d19542e7in July, 2023) implements `--per-test-coverage` (which can also be specified via `lit_config.per_test_coverage`). However, it has a few issues, which the current patch addresses: 1. D154280 implements `--per-test-coverage` only for the case that lit is configured to use an external shell. The current patch extends the implementation to lit's internal shell. 2. In the case that lit is configured to use an external shell, regardless of whether `--per-test-coverage` is actually specified, D154280 causes `%dbg(RUN: at line N)` to be expanded in RUN lines early and in a manner that is specific to sh-like shells. As a result, later code in lit that expands it in a shell-specific manner is useless as there's nothing left to expand. The current patch cleans up the implementation to avoid useless code. 3. Because of issue 2, D154280 corrupts support for windows `cmd` as an external shell (effectively comments out all RUN lines with `:`). The current patch happens to fix that particular corruption by addressing issue 2. However, D122569 (landed in1041a9642bin April, 2022) had already broken support for windows `cmd` as an external shell (discards RUN lines when expanding `%dbg(RUN: at line N)`). The current patch does not attempt to fix that bug. For further details, see the PR discussion of the current patch. The current patch addresses the above issues by implementing `--per-test-coverage` before selecting the shell (internal or external) and by leaving `%dbg(RUN: at line N)` unexpanded there. Thus, it is expanded later in a shell-specific manner, as before D154280. This patch introduces `buildPdbgCommand` into lit's implementation to encapsulate the process of building (or rebuilding in the case of the `--per-test-coverage` implementation) a full `%dbg(RUN: at line N) cmd` line and asserting that the result matches `kPdbgRegex`. It also cleans up that and all other uses of `kPdbgRegex` to operate on the full line with `re.fullmatch` not `re.match`. This change better reflects the intention in every case, but it is expected to be NFC because `kPdbgRegex` ends in `.*` and thus avoids the difference between `re.fullmatch` and `re.match`. The only caveat is that `.*` does not match newlines, but RUN lines cannot contain newlines currently, so this caveat currently shouldn't matter in practice. The original `--per-test-coverage` implementation avoided accumulating `export LLVM_PROFILE_FILE={profile}` insertions across retries (due to `ALLOW_RETRIES`) by skipping the insertion if `%dbg(RUN: at line N)` was not present and thus had already been expanded. However, the current patch makes sure the insertions also happen for commands without `%dbg(RUN: at line N)`, such as preamble commands or some commands from other lit test formats. Thus, the current patch implements a different mechanism to avoid accumulating those insertions (see code comments).
===============================
lit - A Software Testing Tool
===============================
About
=====
*lit* is a portable tool for executing LLVM and Clang style test suites,
summarizing their results, and providing indication of failures. *lit* is
designed to be a lightweight testing tool with as simple a user interface as
possible.
Features
========
* Portable!
* Flexible test discovery.
* Parallel test execution.
* Support for multiple test formats and test suite designs.
Documentation
=============
The official *lit* documentation is in the man page, available online at the LLVM
Command Guide: http://llvm.org/cmds/lit.html.
Source
======
The *lit* source is available as part of LLVM, in the LLVM source repository:
https://github.com/llvm/llvm-project/tree/main/llvm/utils/lit
Contributing to lit
===================
Please browse the issues labeled *tools:llvm-lit* in LLVM's issue tracker for
ideas on what to work on:
https://github.com/llvm/llvm-project/labels/tools%3Allvm-lit
Before submitting patches, run the test suite to ensure nothing has regressed::
# From within your LLVM source directory.
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
Note that lit's tests depend on ``not`` and ``FileCheck``, LLVM utilities.
You will need to have built LLVM tools in order to run lit's test suite
successfully.
You'll also want to confirm that lit continues to work when testing LLVM.
Follow the instructions in http://llvm.org/docs/TestingGuide.html to run the
regression test suite:
make check-llvm
And be sure to run the llvm-lit wrapper script as well:
/path/to/your/llvm/build/bin/llvm-lit utils/lit/tests
Finally, make sure lit works when installed via setuptools:
python utils/lit/setup.py install
lit --path /path/to/your/llvm/build/bin utils/lit/tests