After https://github.com/llvm/llvm-project/pull/80007 Fuchsia builds are
now always building cxx_shared for arm64 and x64 Linux. Ultimately, this
is because the LIBCXX_ENABLE_SHARED is not used in compiler-rt to select
the correct libc++ target, and because cxx_shared is now always defined,
it is selected as a dependency when building runtimes tests.
---------
Co-authored-by: Petr Hosek <phosek@google.com>
I saw cases that this finised before finding `BINGO`, possibly
insufficient number of iteration. In my case, 11,067,133 satisfied.
So, increase the number for now. This change may increase the duration
of this in failing (`BINGO` not found) case.
This patch adds `REQUIRES: shell` to the `focus-function.test` because
the lit internal shell does not support the for loop syntax. This will
make the test file unsupported when running llvm-lit with its internal
shell implementation, which is enabled by turning on the
`LIT_USE_INTERNAL_SHELL=1`.
fixes: #106111
This patch addresses an issue where the `LIBFUZZER_OOP_TARGET`
environment variable was causing "command not found" errors in the
`out-of-process-fuzz` test when running with the
`LIT_USE_INTERNAL_SHELL=1 ninja check-compiler-rt` command.
Error displayed:
```
# .---command stderr------------
# | 'LIBFUZZER_OOP_TARGET=./oop-target > /dev/null 2>&1 ': command not found
# `-----------------------------
# error: command failed with exit status: 127
```
The lit internal shell was not correctly interpreting the command
redirection and environment variable assignment in a single line,
leading to the failure.
In this patch the test was updated to use `env` for setting the
`LIBFUZZER_OOP_TARGET` environment variable. This change ensures that
the command is properly recognized and executed within the internal
shell environment. The output of the fuzzing process was captured in a
temporary file instead of redirecting it directly to `/dev/null`,
allowing for more controlled output management. Finally, the results
from running `oop-target` on all corpus files are now appended to ensure
that every output is thoroughly checked by FileCheck. The `RUN` line
sets the `LIBFUZZER_OOP_TARGET` using `env`, runs the `oop-fuzzer` with
specific options and uses the seed files from `OOP_CORPUS/seed` to
initiate the fuzzing process.
This change is relevant for enabling the lit internal shell by default,
as outlined in [[RFC] Enabling the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
This patch adds the `REQUIRES: shell` directive to six test files that
use the `ulimit` command, ensuring these tests are only run in
environments where a full POSIX-compliant shell is available. The lit
internal shell does not use or support the `ulimit` command, which
causes failures when running tests with `LIT_USE_INTERNAL_SHELL=1 ninja
check-compiler-rt`
Specifically, one of the errors encountered is:
```
# RUN: at line 4
ulimit -s 1000
# executed command: ulimit -s 1000
# .---command stderr------------
# | 'ulimit': command not found
# `-----------------------------
# error: command failed with exit status: 127
```
Since, the lit internal shell doesn't support `ulimit`, adding this
requirement prevents these tests from failing in the lit internal shell,
thereby improving the reliability of the test suite in environments
where the full shell is not available.
This change is relevant for [[RFC] Enabling the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179/3)
fixes: #102398
This patch rewrites tests to remove the use of the `unset` command,
which is not supported in the lit internal shell. The tests now use the
`env -u` to unset environment variables.
The `unset` command is used in shell environments to remove the
environment variable. However, because the lit internal shell does not
support the `unset` command, using it in tests would result in errors or
other unexpected behavior. To overcome this limitation, the tests have
been updated to use the `env -u` command instead. `env -u` is supported
by lit and effectively removes specified environment variables. This
allows the tests to achieve the same goal of unsetting environment
variables while ensuring compatibility with the lit internal shell.
This change is relevant for [[RFC] Enabling the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179/3)
Fixes: #102397
This patch makes the features_dir.test file require a shell when
running. This will make the test file unsupported when running llvm-lit
with its internal shell implementation, which is enabled by turning on
the LIT_USE_INTERNAL_SHELL environment variable. Lit's internal shell
currently does not support for-loop syntax.
This patch adds the `REQUIRES: shell` directive to compiler-rt's fuzzer
tests that require full shell functionality when using the lit internal
shell. These tests depend on features such as background processes,
signal handling, and session management, which are not supported by
lit's internal shell. The addition of this directive ensures that these
tests are only executed in environments that provide the necessary shell
capabilities.
**Details of the Change:**
The following considerations were addressed:
- **Background Processes (`&`):** The tests run commands in the
background using the `&` operator, which allows the shell to execute
commands concurrently without waiting for each one to finish. In a
standard Unix-like shell, this is a common feature that facilitates
multitasking. However, lit's internal shell does not fully support
background job control, which can lead to unpredictable behavior or test
failures. Without proper handling of background processes, subsequent
commands that depend on the status of these processes may not function
correctly.
- **Signal Handling (`kill -SIGUSR1`, `kill -SIGUSR2`, `kill
-SIGINT`):** These tests involve sending specific signals like
`SIGUSR1`, `SIGUSR2`, and `SIGINT` to running processes. These signals
are used to trigger certain behaviors in the processes, such as pausing,
resuming, or terminating. However, lit's internal shell may not handle
these signals properly—it might not send the signal correctly, or the
process might not respond as it should. This could lead to the test
failing, not because the process is incorrect, but because the shell
didn't manage the signals as required.
- **Session Management (`setsid`):** The tests use `setsid` to create
new sessions, detaching processes from their controlling terminal and
isolating them into their own process groups. However, the internal
shell in lit does not support session management features like `setsid`,
this can't correctly isolate and manage processes as required by these
tests.
This change is relevant for enabling the lit internal shell by default,
as outlined in [[RFC] Enabling the Lit Internal Shell by
Default](https://discourse.llvm.org/t/rfc-enabling-the-lit-internal-shell-by-default/80179)
1. Move checks into parent test/CMakeLists.txt
2. COMPILER_RT_INCLUDE_TESTS disable both lit and
gtests. Before it was very inconsistent between
sanitizers.
Disables LSan in order to remove a 1% flake rate in this test.
There's some logic in LeakSanitizer and its integration into libFuzzer
that will disable LSan and re-run the input. This only happens when more
malloc()s are detected than free()s. Under high system load, this
appears to be possible as the "more mallocs than frees" is dependent on
walltime. In these instances, the number of runs ends up being `n + 1`,
which is undesirable.
Update the folder titles for targets in the monorepository that have not
seen taken care of for some time. These are the folders that targets are
organized in Visual Studio and XCode
(`set_property(TARGET <target> PROPERTY FOLDER "<title>")`)
when using the respective CMake's IDE generator.
* Ensure that every target is in a folder
* Use a folder hierarchy with each LLVM subproject as a top-level folder
* Use consistent folder names between subprojects
* When using target-creating functions from AddLLVM.cmake, automatically
deduce the folder. This reduces the number of
`set_property`/`set_target_property`, but are still necessary when
`add_custom_target`, `add_executable`, `add_library`, etc. are used. A
LLVM_SUBPROJECT_TITLE definition is used for that in each subproject's
root CMakeLists.txt.
Currently, the testsuite uses the default runtimes path to find the
runtimes libraries which may or may not match the just-built runtimes.
This change uses the `-resource-dir` flag for clang whenever
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` is set to ensure that we are
actually testing the currently built libraries rather than the ones
bundled with `${COMPILER_RT_TEST_COMPILER}`.
The existing logic works fine when clang and compiler-rt share the same
build directory ``-DLLVM_ENABLE_PROJECTS=clang;compiler-rt`, but when
building compiler-rt separately we need to tell the compiler used for
the tests where it can find the just-built libraries.
This reduces the fixes check-all failures to one in my configuration:
```
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -G Ninja
-DCMAKE_C_COMPILER=$HOME/output/upstream-llvm/bin/clang
-DCMAKE_CXX_COMPILER=$HOME/output/upstream-llvm/bin/clang++
-DCOMPILER_RT_INCLUDE_TESTS=ON
-DLLVM_EXTERNAL_LIT=$HOME/build/upstream-llvm-project-build/bin/llvm-lit
-DLLVM_CMAKE_DIR=$HOME/output/upstream-llvm
-DCOMPILER_RT_DEBUG=OFF
-S $HOME/src/upstream-llvm-project/compiler-rt
-B $HOME/src/upstream-llvm-project/compiler-rt/cmake-build-all-sanitizers
```
This relands the previous PR with fixes for Windows.
Depends on https://github.com/llvm/llvm-project/pull/88074 to be merged
first for GCC buildbots.
Pull Request: https://github.com/llvm/llvm-project/pull/88075
Currently, the testsuite uses the default runtimes path to find the
runtimes libraries which may or may not match the just-built runtimes.
This change uses the `-resource-dir` flag for clang whenever
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` is set to ensure that we are
actually testing the currently built libraries rather than the ones
bundled with `${COMPILER_RT_TEST_COMPILER}`.
The existing logic works fine when clang and compiler-rt share the same
build directory ``-DLLVM_ENABLE_PROJECTS=clang;compiler-rt`, but when
building compiler-rt separately we need to tell the compiler used for
the tests where it can find the just-built libraries.
This reduces the fixes check-all failures to one in my configuration:
```
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -G Ninja
-DCMAKE_C_COMPILER=$HOME/output/upstream-llvm/bin/clang
-DCMAKE_CXX_COMPILER=$HOME/output/upstream-llvm/bin/clang++
-DCOMPILER_RT_INCLUDE_TESTS=ON
-DLLVM_EXTERNAL_LIT=$HOME/build/upstream-llvm-project-build/bin/llvm-lit
-DLLVM_CMAKE_DIR=$HOME/output/upstream-llvm
-DCOMPILER_RT_DEBUG=OFF
-S $HOME/src/upstream-llvm-project/compiler-rt
-B $HOME/src/upstream-llvm-project/compiler-rt/cmake-build-all-sanitizers
```
Reviewed By: vitalybuka, delcypher, MaskRay
Pull Request: https://github.com/llvm/llvm-project/pull/83088
The HOST_LINK_VERSION is a hardcoded string in Darwin clang that detects
the linker version at configure time. The driver uses this information
to build the correct set of arguments for the linker. This patch detects
the linker version again during compiler-rt configuration and passes it
to the libfuzzer tests. This allows a clang built on a machine with a
new linker to run compiler-rt tests on a machine with an old linker.
rdar://125932376
With https://github.com/llvm/llvm-project/pull/83088, we now need the
runtimes to be built before running test if
COMPILER_RT_TEST_STANDALONE_BUILD_LIBS is true, since otherwise we
get failures running `ninja check-all` such as the following:
```
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.fuzzer-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-basic-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-fdr-x86_64.a: No such file or directory
```
This is a follow-up to 058e9b03 which started removing these checks
and it should make it easier to stop forcing COMPILER_RT_STANDALONE_BUILD
for runtimes builds in the future.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83651
The FileCheck string `LLVMFuzzerCustomMutatorLongSequence: {{.*}} MS:
{{[0-9]*}} {{(([a-zA-Z]*-){11,})}} {{.*}}` is too restrictive and may
fail the test in some case.
If we look at the commit that added this
check(66df98945e),
This check is for printing out the long mutation sequence, such as this
one
```
#53552 REDUCE cov: 6 ft: 6 corp: 5/9b lim: 4096 exec/s: 0 rss: 37Mb L: 2/3 MS: 54 ChangeByte-PersAutoDict-ChangeBit-ChangeBinInt-ChangeBit-ChangeBit-ChangeByte-CMP-EraseBytes-EraseBytes-CrossOver-InsertRepeatedBytes-ChangeByte-EraseBytes-InsertRepeatedBytes-ShuffleBytes-ChangeByte-ShuffleBytes-ChangeBit-CrossOver-ChangeBit-ShuffleBytes-ChangeBinInt-ShuffleBytes-EraseBytes-InsertByte-Custom-ShuffleBytes-CopyPart-InsertRepeatedBytes-PersAutoDict-InsertRepeatedBytes-ChangeByte-CrossOver-CrossOver-PersAutoDict-PersAutoDict-EraseBytes-ChangeBit-CopyPart-ChangeByte-CopyPart-InsertRepeatedBytes-CrossOver-CrossOver-CrossOver-CrossOver-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-EraseBytes-Custom- DE: "\377\377"-"\001\000"-"\001\000"-"\000\000\000\000\000\000\000\000"-"\001\000\000\000"-
```
But if we look at the code doing the printing
```cpp
void MutationDispatcher::PrintMutationSequence(bool Verbose) {
Printf("MS: %zd ", CurrentMutatorSequence.size());
size_t EntriesToPrint =
Verbose ? CurrentMutatorSequence.size()
: std::min(kMaxMutationsToPrint, CurrentMutatorSequence.size());
for (size_t i = 0; i < EntriesToPrint; i++)
Printf("%s-", CurrentMutatorSequence[i].Name);
if (!CurrentDictionaryEntrySequence.empty()) {
Printf(" DE: ");
EntriesToPrint = Verbose ? CurrentDictionaryEntrySequence.size()
: std::min(kMaxMutationsToPrint,
CurrentDictionaryEntrySequence.size());
for (size_t i = 0; i < EntriesToPrint; i++) {
Printf("\"");
PrintASCII(CurrentDictionaryEntrySequence[i]->GetW(), "\"-");
}
}
}
```
We can see that the `DE: XXX` is not always printed. So the following
output is possible(and is from real-life failure), notince the missing
of `DE: XXX`.
```
#13613 NEW cov: 5 ft: 5 corp: 4/6b lim: 4096 exec/s: 0 rss: 32Mb L: 2/2 MS: 27 InsertByte-ChangeBinInt-ChangeBinInt-CrossOver-ShuffleBytes-ChangeBit-EraseBytes-ShuffleBytes-InsertByte-InsertRepeatedBytes-CopyPart-InsertByte-ChangeByte-ChangeBit-InsertByte-CrossOver-EraseBytes-CopyPart-ShuffleBytes-EraseBytes-InsertByte-InsertRepeatedBytes-CrossOver-CrossOver-ShuffleBytes-ChangeBit-Custom-
#13765 ......
```
This output is totally legit and will fail that check.
So I remove the check for the following strings, I think `MS: {{[0-9]*}}
{{(([a-zA-Z]*-){11,})}}` is sufficient for checking the long mutation
sequence. This should help resolve the flaky failure of
fuzzer-custommutator.test.
Add config name for fuzzer lit test, to make it easier to identify failures are with which config.
Before this change, same lit tests with different configs will share the same test name.
```
********************
Failed Tests (2):
libFuzzer :: fuzzer-flags.test
libFuzzer :: fuzzer-flags.test
```
Actually this is a failure of two lit tests(two configs of the same test).
With this change, the names will be different.
```
********************
Failed Tests (2):
libFuzzer-i386-default-Linux ::fuzzer-flags.test
libFuzzer-x86_64-default-Linux :: fuzzer-flags.test
```
Reviewed By: MaskRay, vitalybuka
Differential Revision: https://reviews.llvm.org/D158696
The test requires that an integer division by zero causes a trap, leading to a
signal like SIGFPE. This is the case on x86. On many other architectures, such
as Arm (unless software division is used and `__aeabi_idiv0` is patched),
Power, RISC-V, LoongArch, there is no trap. Therefore, it's more appropriate to
invert the condition to run the test.
Reviewed By: Ami-zhang
Differential Revision: https://reviews.llvm.org/D155644
This is an ongoing series of commits that are reformatting our
Python code. This catches the last of the python files to
reformat. Since they where so few I bunched them together.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: jhenderson, #libc, Mordante, sivachandra
Differential Revision: https://reviews.llvm.org/D150784
This test is heavy on test resources and involves moving a large 2GB+ file
across an ssh connection when testing on remote devices. Exclude all tests
except for macOS testing (on host). Remote device testing for macOS may
eventually cause a problem on macOS as well w/o better handling of tests
that require more resource/time than we currently handle.
Introduced by:
https://reviews.llvm.org/D146189
Subsequent exclusion of all but darwin by:
https://reviews.llvm.org/D147094
Subsequent exclusion of tvOS and watchOS by:
https://reviews.llvm.org/D147502
Subsequent exclusion of all but macOS...
rdar://107570309
Reviewed By: yln, thetruestblue
Differential Revision: https://reviews.llvm.org/D148727
The test "big-file-copy.test" introduced in D146189 and constrained to darwin by
D147094, is by this differential further constrained to only those devices with
sufficient resources. Also correct the test to read the environment variable
"result" from the same shell in which it was stored (which may differ on devices).
Reviewed By: thetruestblue
Differential Revision: https://reviews.llvm.org/D147502
This test has to be limited to darwin due to multiple failures on other
platforms for multple reasons. (Timeout, puts() limit, etc.). This
commit modifies D146189.
Reviewed By: NoQ
Differential Revision: https://reviews.llvm.org/D147094
CopyFileToErr() uses Printf("%s", ...) which fails with a negative size on
files >2Gb (Its path is through var-args wrappers to an unnecessary "%s"
expansion and subject to int overflows) Using puts() in place of printf()
bypasses this path and writes the string directly to stderr. This avoids the
present loss of data when a crashed worker has generated >2Gb of output.
rdar://99384640
Reviewed By: yln, rsundahl
Differential Revision: https://reviews.llvm.org/D146189
There is test for ft file-id in control file, but no test for cov line.
Without the test, a invalid cov file-id would cause crash.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D145672
CopyFileToErr() uses Printf("%s", ...) which fails with a negative size on
files >2Gb (Its path is through var-args wrappers to an unnecessary "%s"
expansion and subject to int overflows) Using puts() in place of printf()
bypasses this path and writes the string directly to stderr. This avoids the
present loss of data when a crashed worker has generated >2Gb of output.
rdar://99384640
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D146189
`CollectDataFlow()` uses `Flags.collect_data_flow` and
`Flags.data_flow_trace` at the same time. But in the null check before
the invocation, only `Flags.collect_data_flow` is checked, and there is
no other method to make sure `Flags.data_flow_trace` is not null, so
adding a null check for `Flags.data_flow_trace`.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D145040
These four tests are failing on tvOS devices (not simulators) so XFAIL
them for now for CI and investigate further.
rdar://99981102
Differential Revision: https://reviews.llvm.org/D133963
The fuzzer tests cross_over.test and merge-control-file.test are not handled
correctly on ios device testing. On-device testing requires the macros %t, %s,
etc. to be expanded for a different default directory than when testing on host.
rdar://99889376
Differential Revision: https://reviews.llvm.org/D133811
With this change, fuzz targets may choose to return -1
to indicate that the input should not be added to the corpus
regardless of the coverage it generated.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D128749
As with Linux placce the Counters array in the __libfuzzer_extra_counters
section. This fixes the test on FreeBSD.
Reviewed by: vitalybuka
Differential Revision: https://reviews.llvm.org/D125902
3bd112c720 fixed the fuzzing test on Linux, which, after
https://reviews.llvm.org/D125933, has one less branch. Turns out, on
Windows, that it still has the extra branch. I'm guessing that's because
exit() isn't known to be noreturn on Windows or something.
Either way, just make the test more tolerant.
https://reviews.llvm.org/D125933 improved some of LLVM's handling of
binary ORs, which meant we have one less conditional branch, because the
'if (Size > 5 && Data[5] == 'R')' and 'if (bits == 63)' branches are now
correctly folded.
