Unconditionally change std::string's alignment to 8.
This change saves memory by providing the allocator more freedom to
allocate the most
efficient size class by dropping the alignment requirements for
std::string's
pointer from 16 to 8. This changes the output of std::string::max_size,
which makes it ABI breaking.
That said, the discussion concluded that we don't care about this ABI
break. and would like this change enabled universally.
The ABI break isn't one of layout or "class size", but rather the value
of "max_size()" changes, which in turn changes whether `std::bad_alloc`
or `std::length_error` is thrown for large allocations.
This change is the child of PR #68807, which enabled the change behind
an ABI flag.
This is the last PR that's needed (for now) to get libc++'s tests
working with MSVC's STL.
The ADDITIONAL_COMPILE_FLAGS machinery is very useful, but also very
problematic for MSVC, as it doesn't understand most of Clang's compiler
options. We've been dealing with this by simply marking anything that
uses ADDITIONAL_COMPILE_FLAGS as FAIL or SKIPPED, but that creates
significant gaps in test coverage.
Fortunately, ADDITIONAL_COMPILE_FLAGS also supports "features", which
can be slightly enhanced to send Clang-compatible and MSVC-compatible
options to the right compilers.
This patch adds the gcc-style-warnings and cl-style-warnings Lit features,
and uses that to pass the appropriate warning flags to tests. It also uses
TEST_MEOW_DIAGNOSTIC_IGNORED for a few local suppressions of MSVC
warnings.
This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.
**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.
To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.
Originally proposed here: https://reviews.llvm.org/D132769
Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214
This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.
The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.
This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.
Due to the functionality introduced in
[D132522](dd1b7b797a),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.
If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com
Extend `std::basic_string` tests to cover more buffer situations and
length in general, particularly non-SSO cases after SSO test cases
(changing buffers). This commit is a side effect of working on tests for
ASan annotations.
Related PR: https://github.com/llvm/llvm-project/pull/72677
Found while running libc++'s tests with MSVC's STL. This fixes 3 kinds of warnings:
- Add void-casts to fix `-Wunused-variable` warnings.
- Avoid sign/truncation warnings in `ConvertibleToIntegral.h`.
- Add `TEST_STD_AT_LEAST_23_OR_RUNTIME_EVALUATED` to avoid mixing preprocessor
and runtime tests.
- Cleanup: Add `TEST_STD_AT_LEAST_20_OR_RUNTIME_EVALUATED` for
consistency.
Picolibc is a C Standard Library that is commonly used in embedded
environments. This patch adds initial support for this configuration
along with pre-commit CI. As of this patch, the test suite only builds
the tests and nothing is run. A follow-up patch will make the test suite
actually run the tests.
Differential Revision: https://reviews.llvm.org/D154246
I've structured this into a series of commits for even easier reviewing,
if that helps. I could easily split this up into separate PRs if
desired, but as this is low-risk with simple edits, I thought one PR
would be easiest.
* Drop unnecessary semicolons after function definitions.
* Cleanup comment typos.
* Cleanup `static_assert` typos.
* Cleanup test code typos.
+ There should be no functional changes, assuming I've changed all
occurrences.
* ~~Fix massive test code typos.~~
+ This was a real problem, but needed more surgery. I reverted those
changes here, and @philnik777 is fixing this properly with #73444.
* clang-formatting as requested by the CI.
Found while running libc++'s test suite with MSVC's STL.
This is structured into a series of commits for easier reviewing; I
could also split this into smaller PRs if desired.
* Add void-casts for `invoke_r` calls to fix MSVC STL `[[nodiscard]]`
warnings.
+ Our rationale is that if someone is calling `invoke_r<NonVoidType>`,
it sure looks like they care about the return value.
* Add `[[maybe_unused]]` to silence `-Wunused-parameter` warnings.
+ This happens because the parameters are used within `LIBCPP_ASSERT`,
which vanishes for MSVC's STL. This also motivates the following
changes.
* Add `[[maybe_unused]]` to fix `-Wunused-variable` warnings.
* Always void-cast `debug_comparisons` to fix `-Wunused-variable`
warnings.
+ As this was already unused with a void-cast in one
`_LIBCPP_HARDENING_MODE` branch, I'm simply lifting it next to the
variable definition.
* Add `[[maybe_unused]]` to fix `-Wunused-local-typedef` warnings.
If we know that index is larger than SSO size, we know that we can't be
in SSO case, and should access the pointer. This removes extra check
from operator[] for inputs known at compile time to be larger than SSO.
This updates the clang-format we use in libc++ to 17. This is necessary
to start running the generated-files checks in GitHub Actions (in
#68920). In fact this is a pre-existing issue regardless of #68920 --
right now our ignore_format.txt job disagrees with the LLVM-wide
clang-format job.
While doing this, I also found a few tests that were either clearly
incorrect (e.g. testing the wrong function) or that lacked basic test
coverage like testing std::string itself (e.g. the test was only checking
std::basic_string with a custom allocator). In these cases, I did a few
conservative drive-by changes.
Differential Revision: https://reviews.llvm.org/D140550
Co-authored-by: Brendan Emery <brendan.emery@esrlabs.com>
Previously, assignment to a std::basic_string type with a _custom_
allocator could under certain conditions attempt to interpret part of
the target string's "short" string-content as if it was a "long" data
pointer, and attempt to deallocate a garbage value.
This is a serious bug, but code in which it might happen is rare. It
required:
1. the basic_string must be using a custom allocator type which sets the
propagate_on_container_copy_assignment trait to true (thus, it does not
affect the default allocator, nor most custom allocators).
2. the allocator for the target string must compare not equal to the
allocator for the source string (many allocators always compare equal).
3. the source of the copy must currently contain a "long" string, and
the assignment-target must currently contain a "short" string.
Finally, the issue would've typically been innocuous when the bytes
misinterpreted as a pointer were all zero, as deallocating a nullptr is
typically a no-op. This is why existing test cases did not exhibit an
issue: they were all zero-length strings, which do not have data in the
bytes interpreted as a pointer.
- add the `from_range_t` constructors and the related deduction guides;
- add the `insert_range`/`assign_range`/etc. member functions.
(Note: this patch is split from https://reviews.llvm.org/D142335)
Differential Revision: https://reviews.llvm.org/D149832
I made sure they all had some expected-error output in them. Many of
these tests would be better implemented as a positive test using SFINAE,
but this is beyond the scope of this patch.
Differential Revision: https://reviews.llvm.org/D153980
Some tests in our test suite are unbelievably slow on GCC due to the
use of the always_inline attribute. See [1] for more details.
This patch introduces the GCC-ALWAYS_INLINE-FIXME lit feature to
disable tests that are plagued by that issue. At the same time, it
moves several existing tests from ad-hoc `UNSUPPORTED: gcc-12` markup
to the new GCC-ALWAYS_INLINE-FIXME feature, and marks the slowest tests
reported by the CI as `UNSUPPORTED: GCC-ALWAYS_INLINE-FIXME`.
[1]: https://discourse.llvm.org/t/rfc-stop-supporting-extern-instantiations-with-gcc/71277/1
Differential Revision: https://reviews.llvm.org/D152736
This is an ongoing series of commits that are reformatting our
Python code.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: #libc, kwk, Mordante
Differential Revision: https://reviews.llvm.org/D150763
During the ISO C++ Committee meeting plenary session the C++23 Standard
has been voted as technical complete.
This updates the reference to c++2b to c++23 and updates the __cplusplus
macro.
Note since we use clang-tidy 16 a small work-around is needed. Clang
knows -std=c++23 but clang-tidy not so for now force the lit compiler
flag to use -std=c++2b instead of -std=c++23.
Reviewed By: #libc, philnik, jloser, ldionne
Differential Revision: https://reviews.llvm.org/D150795
We plan to add concepts for checking that iterators actually provide what they claim to. This is to avoid people thinking that these type traits actually check the iterator requirements in more detail.
Reviewed By: ldionne, #libc
Spies: Mordante, libcxx-commits, wenlei
Differential Revision: https://reviews.llvm.org/D150801
In libc++'s `wchar.h`, before we forward to the system `wchar.h`, we set `__CORRECT_ISO_CPP_WCHAR_H_PROTO` to ensure it defines the correct signature (e.g. `extern "C++" const wchar_t *wmemchr` and not `extern wchar_t *wmemchr`). After D148542, there are cases where we include the system `wchar.h` from within `__mbstate_t.h` without setting that, and so we get a function type mismatch if we transitively include `wchar.h` through multiple headers in a modules-enabled build. Consistently setting it here resolves those build errors.
Alternative 1: we could put this in `__config` instead. I chose to put it here for a more limited scope.
Alternative 2: we could patch `wchar.h` itself to work correctly and remove references `__CORRECT_ISO_CPP_WCHAR_H_PROTO` from libc++ entirely. It does already set it, but with an additional condition that it is being built by GCC >= 4.4. Clang does pretend to be GCC via `__GNUC__` etc. which can be controlled via `-fgnuc-version` command line flags, but that might have other consequences.
Reviewed By: ldionne, #libc, MaskRay
Differential Revision: https://reviews.llvm.org/D150015
These test cases were fixed with AIX 73TL1, and are currently passing on AIX machines with that fix. This fix has also been backported to the 7.2 service line. These were tested on a machine with AIX 7.2 TL 5 SP4 installed.
Differential Revision: https://reviews.llvm.org/D148040
The use_system_cxx_lib Lit feature was only used for back-deployment
testing. However, one immense hole in that setup was that we didn't
have a proper way to test Apple's own libc++ outside of back-deployment,
which was embodied by the fact that we needed to define _LIBCPP_DISABLE_AVAILABILITY
when testing (see change in libcxx/utils/libcxx/test/params.py).
This led to the apple-system testing configuration not checking for
availability markup, which is obviously quite bad since the library
we ship actually has availability markup.
Using stdlib=<VENDOR>-libc++ instead to encode back-deployment restrictions
on tests is simpler and it makes it possible to naturally support tests
such as availability markup checking even in the tip-of-trunk Apple-libc++
configuration.
Differential Revision: https://reviews.llvm.org/D146366
This has been done using the following commands
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)ptrdiff_t)|\1std::\2|' \{} \;
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)max_align_t)|\1std::\2|' \{} \;
The std module doesn't export declarations in the global namespaace.,
This is a preparation for that module.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D146550
This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)size_t)|\1std::\2|' \{} \;
And manually removed some false positives in std/depr/depr.c.headers.
The `std` module doesn't export `::size_t`, this is a preparation for that module.
Reviewed By: ldionne, #libc, EricWF, philnik
Differential Revision: https://reviews.llvm.org/D146088
We pretty consistently don't define those cause they are not needed,
and it removes the potential pitfall to think that these tests are
being run. This doesn't touch .compile.fail.cpp tests since those
should be replaced by .verify.cpp tests anyway, and there would be
a lot to fix up.
As a fly-by, I also fixed a bit of formatting, removed a few unused
includes and made some very minor, clearly NFC refactorings such as
in allocator.traits/allocator.traits.members/allocate.verify.cpp where
the old test basically made no sense the way it was written.
Differential Revision: https://reviews.llvm.org/D146236
This has been done using the following command
find libcxx/test -type f -exec perl -pi -e 's|^([^/]+?)((?<!::)(?<!::u)u?int(_[a-z]+)?[0-9]{1,2}_t)|\1std::\2|' \{} \;
And manually removed some false positives in std/depr/depr.c.headers.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D145880
Avoids using operator& in basic_string since an evil char-like type can
hijack this operator. Added some more evil operators, this found a place
where equality was compared directly and not via the traits.
This adds a helper test string. This is now only used in a few tests,
but the intention is to use this in more tests for basic_string.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D145257
First, fix a collision with the Point type from MacTypes.h, which was
reported on Slack, 2022-07-31: https://cpplang.slack.com/archives/C2X659D1B/p1659284691275889
Second, rename the meta:: namespace to types::. OSX's "/usr/include/ncurses.h"
defines a `meta` function, and is (for some reason) included in
"<SDK>/usr/include/module.modulemap", so that identifier is off-limits
for us to use in anything that compiles with -fmodules:
libcxx/test/support/type_algorithms.h:16:11: error: redefinition of 'meta' as different kind of symbol
namespace meta {
^
<SDK>/usr/include/ncurses.h:603:28: note: previous definition is here
extern NCURSES_EXPORT(int) meta (WINDOW *,bool); /* implemented */
^
Finally, add a CI configuration for modules on OS X to make sure it
does not regress.
Differential Revision: https://reviews.llvm.org/D144915
The `basic_string_view` constructor accepting a contiguous range rejects
converting between `basic_string_view` even when only the trait types vary.
This prevents conversions for converting from `basic_string_view<C, T1>` and
`basic_string<C, T1, A>` to `basic_string_view<C, T2>`. Recently, this
constructor was made `explicit`, so there's no reason to really forbid this
conversion anymore.
Relax the restriction that the trait types need to match in this constructor.
Differential Revision: https://reviews.llvm.org/D143972
When support for declaring the c8rtomb() and mbrtoc8() functions within the
std namespace was added in commit 7e7013c5d4,
internal glibc macros were used to determine if C2X extensions are enabled.
Specifically, a check for whether `__GLIBC_USE` is defined and whether
`__GLIBC_USE(ISOC2X)` is non-0 was added. `__GLIBC_USE` is an internal
detail of the glibc implementation that may be changed or removed in the
future potentially leading to inconsistency or compilation failures. This
change removes the use of the internal glibc macro to avoid such problems.
Unfortunately, without another mechanism to determine if C2X extensions are
enabled, this removal will result in inconsistent declarations of the
c8rtomb() and mbrtoc8() functions; when C++ char8_t support is not enabled, but
C2X extensions are, these functions will be declared in the global namespace
but not in the std namespace. This situation will improve when C23 support
is finalized and the check can be re-implemented using `__STDC_VERSION__`.
* Using one-or-two letter names for globals is asking for shadowing warnings.
* MSVCSTL's container proxy allocations strike again
* MSVCSTL's `<string>` doesn't define `std::out_of_range`
* `basic_string::substr` takes two arguments of type `size_type`. Let's use that type instead of `size_t` and `ptrdiff_t` to avoid narrowing warnings.
Differential Revision: https://reviews.llvm.org/D141253
The converting constructor is ill-formed, and `==` is missing. (I didn't implement `!=` since the test is C++20-and-later only; I'll let the compiler do it for us.)
Drive-by: change 4-space indent on line 27 to 2-space indent to be consistent with the rest of the test.
Differential Revision: https://reviews.llvm.org/D131079
When we define the const-correct overloads of <string.h> functions in
libc++ itself, use builtins whenever possible. This avoids depending on
the presence of these functions in the C library headers.
Also, as a fly-by, improve the tests for these functions since we
basically didn't check anything but their signature. We could have
used the wrong builtin (as long as the signature matched) without ever
noticing, which was quite scary.
Differential Revision: https://reviews.llvm.org/D138684
This patch removes the base template implementation for std::char_traits.
If my reading of http://eel.is/c++draft/char.traits is correct, the
Standard mandates that the library provides specializations for several
types like char and wchar_t, but not any implementation in the base
template. Indeed, such an implementation is bound to be incorrect for
most types anyways, since things like `eof()` and `int_type` will definitely
have to be customized.
Since the base template implementation should not have worked for anyone,
this shouldn't be a breaking change (I expect that anyone defining a
custom character type today will already have to provide their own
specialization of char_traits). However, since we're aware of some users
of char_traits for unsigned char and signed char, we're keeping those two
specializations around for two releases to give people some time to migrate.
Differential Revision: https://reviews.llvm.org/D138307
We currently call a lot of functions with the same list of types. To avoid forgetting any of them, this patch adds type_lists and utilities for it. Specifically, it adds
- `type_list` - This is just a list of types
- `concatenate` - This allows concatenating type_lists
- `for_each` - Iterate over a type_list
Reviewed By: ldionne, #libc
Spies: jloser, EricWF, libcxx-commits
Differential Revision: https://reviews.llvm.org/D137476
This doesn't affect our ABI because `std::string::substr()` isn't in the dylib and the mangling of `substr() const` and `substr() const&` are different.
Reviewed By: ldionne, Mordante, var-const, avogelsgesang, #libc
Spies: arphaman, huixie90, libcxx-commits
Differential Revision: https://reviews.llvm.org/D131668
