Add LIBC_ASSERT statements to FixedVector implementation, and zero out
the memory when the elements are removed to flag out-of-bound access and
dangling pointer/reference access.
This change unmasks the bug in one of FixedVector uses for atexit
handlers: dangling reference use, which was actually led to crashes in
the wild (with prod blockstore implementation). Fix it in this CL.
Get rid of the following arguments to libc_support_library and
libc_function rules:
* `defines` (for raw_mutex.h) - it wasn't used correctly (e.g. didn't
provide actual value for spin count), and we can instead fallback to
defaults set in the header itself (or rely on library-level configure
options).
* `features` - there's no need to disable sanitization for a subset of
memory functions -- it generally should be the vendor / user
responsibility to control it (e.g. don't include instrumented libc
functions in the build, since they would be provided by sanitizer
runtimes instead).
* `local_defines` (for printf_parser) - no longer needed, since
LIBC_COPT_MOCK_ARG_LIST has been removed in
e0be78be42
This also removes two ad-hoc BUILD rules (strcpy_sanitized and
printf_mock_parser) which are no longer needed and can be replaced by
strcpy and printf_parser, respectively.
Co-authored-by: Alexey Samsonov <samsonov@google.com>
Extract all compiler options used to build "release" versions of libc
API functions into a separate helper function, instead of burying this
logic inside libc_function() macro.
With this change, we further split two "flavors" of cc_library()
produced for each libc public function:
* `<function>.__internal__` library used in unit tests is *not* built
with release copts and is thus indistinguishable from regular
libc_support_library(). Arguably, it's a good thing, because all sources
in a unit test are built with the same set of compiler flags, instead of
"franken-build" when a subset of sources is always built with -O3. If a
user needs to run the tests in optimized mode, they should really be
using Bazel invocation-level compile flags instead.
* `<function>` library that libc users can use to construct their own
static archive *is* built with the same release copts as before. There
is a pre-existing problem that its libc_support_library() dependencies
are not built with the same copts. We're not addressing it here now.
`SupportTests` fails in the bazel macOS sandbox, because
`FileSystemTest.permissions` expects to be able to modify file
permissions on some otherwise protected files.
Previously this test was marked `local` in bazel, which has
additional undesirable effects such as skipping remote build and cache.
Tighten the bazel tags to just `no-sandbox`. Note in particular, that
this allows the test to build, execute, and cache remotely (if
configured).
Testing:
- Verified this test fails (as expected) on macOS with no tags, and
passes with `no-sandbox`.
- Verified this test passes when executed remotely (using an Engflow RBE
setup) with `no-sandbox`.
This is a companion to #118583, although it can be landed independently
because since #117922 dialects do not have to use the same Python
binding framework as the Python core code.
This PR ports all of the in-tree dialect and pass extensions to
nanobind, with the exception of those that remain for testing pybind11
support.
This PR also:
* removes CollectDiagnosticsToStringScope from NanobindAdaptors.h. This
was overlooked in a previous PR and it is duplicated in Diagnostics.h.
---------
Co-authored-by: Jacques Pienaar <jpienaar@google.com>
Relands #118583, with a fix for Python 3.8 compatibility. It was not
possible to set the buffer protocol accessers via slots in Python 3.8.
Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.
For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.
To a large extent, this is a mechanical change, for instance changing
`pybind11::` to `nanobind::`.
Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.
Why? https://nanobind.readthedocs.io/en/latest/why.html says it better
than I can, but my primary motivation for this change is to improve MLIR
IR construction time from JAX.
For a complicated Google-internal LLM model in JAX, this change improves
the MLIR
lowering time by around 5s (out of around 30s), which is a significant
speedup for simply switching binding frameworks.
To a large extent, this is a mechanical change, for instance changing
`pybind11::`
to `nanobind::`.
Notes:
* this PR needs Nanobind 2.4.0, because it needs a bug fix
(https://github.com/wjakob/nanobind/pull/806) that landed in that
release.
* this PR does not port the in-tree dialect extension modules. They can
be ported in a future PR.
* I removed the py::sibling() annotations from def_static and def_class
in `PybindAdapters.h`. These ask pybind11 to try to form an overload
with an existing method, but it's not possible to form mixed
pybind11/nanobind overloads this ways and the parent class is now
defined in nanobind. Better solutions may be possible here.
* nanobind does not contain an exact equivalent of pybind11's buffer
protocol support. It was not hard to add a nanobind implementation of a
similar API.
* nanobind is pickier about casting to std::vector<bool>, expecting that
the input is a sequence of bool types, not truthy values. In a couple of
places I added code to support truthy values during casting.
* nanobind distinguishes bytes (`nb::bytes`) from strings (e.g.,
`std::string`). This required nb::bytes overloads in a few places.
Fixes:
ERROR: The project you're trying to build requires Bazel 7.3.0
(specified
in llvm-project/utils/bazel/.bazelversion), but it wasn't found in
/usr/bin.
You can install the required Bazel version via apt:
sudo apt update && sudo apt install bazel-7.3.0
If this doesn't work, check Bazel's installation instructions for help:
https://bazel.build/install/ubuntu
Link: https://blog.bazel.build/2024/12/09/bazel-8-release.html
