caio/clang-p2996
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[libc++][modules] Rewrite the modulemap to have fewer top-level modules (#107638)

Browse Source 
This patch rewrites the modulemap to have fewer top-level modules.
Previously, our modulemap had one top level module for each header in
the library, including private headers. This had the well-known problem
of making compilation times terrible, in addition to being somewhat
against the design principles of Clang modules.

This patch provides almost an order of magnitude compilation time
improvement when building modularized code (certainly subject to
variations). For example, including <ccomplex> without a module cache
went from 22.4 seconds to 1.6 seconds, a 14x improvement.

To achieve this, one might be tempted to simply put all the headers in a
single top-level module. Unfortunately, this doesn't work because libc++
provides C compatibility headers (e.g. stdlib.h) which create cycles
when the C Standard Library headers are modularized too. This is
especially tricky since base systems are usually not modularized: as far
as I know, only Xcode 16 beta contains a modularized SDK that makes this
issue visible. To understand it, imagine we have the following setup:

   // in libc++'s include/c++/v1/module.modulemap
   module std {
      header stddef.h
      header stdlib.h
   }

   // in the C library's include/module.modulemap
   module clib {
      header stddef.h
      header stdlib.h
   }

Now, imagine that the C library's <stdlib.h> includes <stddef.h>,
perhaps as an implementation detail. When building the `std` module,
libc++'s <stdlib.h> header does `#include_next <stdlib.h>` to get the C
library's <stdlib.h>, so libc++ depends on the `clib` module.

However, remember that the C library's <stdlib.h> header includes
<stddef.h> as an implementation detail. Since the header search paths
for libc++ are (and must be) before the search paths for the C library,
the C library ends up including libc++'s <stddef.h>, which means it
depends on the `std` module. That's a cycle.

To solve this issue, this patch creates one top-level module for each C
compatibility header. The rest of the libc++ headers are located in a
single top-level `std` module, with two main exceptions. First, the
module containing configuration headers (e.g. <__config>) has its own
top-level module too, because those headers are included by the C
compatibility headers.

Second, we create a top-level std_core module that contains several
dependency-free utilities used (directly or indirectly) from the __math
subdirectory. This is needed because __math pulls in a bunch of stuff,
and __math is used from the C compatibility header <math.h>.

As a direct benefit of this change, we don't need to generate an
artificial __std_clang_module header anymore to provide a monolithic
`std` module, since our modulemap does it naturally by construction.

A next step after this change would be to look into whether math.h
really needs to include the contents of __math, and if so, whether
libc++'s math.h truly needs to include the C library's math.h header.
Removing either dependency would break this annoying cycle.

Thanks to Eric Fiselier for pointing out this approach during a recent
meeting. This wasn't viable before some recent refactoring, but wrapping
everything (except the C headers) in a large module is by far the
simplest and the most effective way of doing this.

Fixes #86193
This commit is contained in:
Louis Dionne
2024-09-26 13:19:48 -04:00
committed by GitHub
parent 0215579dab
commit bc6bd3bc1e
8 changed files with 2163 additions and 2329 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
libcxx/include/CMakeLists.txt
View File

@@ -687,7 +687,6 @@ set(files
__ranges/views.h
__ranges/zip_view.h
__split_buffer
__std_clang_module
__std_mbstate_t.h
__stop_token/atomic_unique_lock.h
__stop_token/intrusive_list_view.h

1
libcxx/include/__format/formatter_integral.h
View File

@@ -27,6 +27,7 @@
#include <__type_traits/make_unsigned.h>
#include <__utility/unreachable.h>
#include <array>
#include <cstdint>
#include <limits>
#include <string>
#include <string_view>

193
libcxx/include/__std_clang_module
View File

@@ -1,193 +0,0 @@
// -*- C++ -*-
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
// WARNING, this entire header is generated by
// utils/generate_std_clang_module_header.py
// DO NOT MODIFY!
// This header should not be directly included, it's exclusively to import all
// of the libc++ public clang modules for the `std` clang module to export. In
// other words, it's to facilitate `@import std;` in Objective-C++ and `import std`
// in Swift to expose all of the libc++ interfaces. This is generally not
// recommended, however there are some clients that need to import all of libc++
// without knowing what "all" is.
#if !__building_module(std)
# error "Do not include this header directly, include individual headers instead"
#endif
#include <__config>
#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
# pragma GCC system_header
#endif
#include <algorithm>
#include <any>
#include <array>
#if !defined(_LIBCPP_HAS_NO_ATOMIC_HEADER)
# include <atomic>
#endif
#include <barrier>
#include <bit>
#include <bitset>
#include <cassert>
#include <ccomplex>
#include <cctype>
#include <cerrno>
#include <cfenv>
#include <cfloat>
#include <charconv>
#include <chrono>
#include <cinttypes>
#include <ciso646>
#include <climits>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <clocale>
#endif
#include <cmath>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <codecvt>
#endif
#include <compare>
#include <complex.h>
#include <complex>
#include <concepts>
#include <condition_variable>
#include <coroutine>
#include <csetjmp>
#include <csignal>
#include <cstdarg>
#include <cstdbool>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <ctgmath>
#include <ctime>
#include <ctype.h>
#include <cuchar>
#include <cwchar>
#include <cwctype>
#include <deque>
#include <errno.h>
#include <exception>
#include <execution>
#include <expected>
#include <experimental/iterator>
#include <experimental/memory>
#include <experimental/propagate_const>
#include <experimental/simd>
#include <experimental/type_traits>
#include <experimental/utility>
#include <fenv.h>
#include <filesystem>
#include <float.h>
#include <format>
#include <forward_list>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <fstream>
#endif
#include <functional>
#include <future>
#include <initializer_list>
#include <inttypes.h>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <iomanip>
#endif
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <ios>
#endif
#include <iosfwd>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <iostream>
#endif
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <istream>
#endif
#include <iterator>
#include <latch>
#include <limits>
#include <list>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <locale.h>
#endif
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <locale>
#endif
#include <map>
#include <math.h>
#include <mdspan>
#include <memory>
#include <memory_resource>
#include <mutex>
#include <new>
#include <numbers>
#include <numeric>
#include <optional>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <ostream>
#endif
#include <print>
#include <queue>
#include <random>
#include <ranges>
#include <ratio>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <regex>
#endif
#include <scoped_allocator>
#include <semaphore>
#include <set>
#include <shared_mutex>
#include <source_location>
#include <span>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <sstream>
#endif
#include <stack>
#if !defined(_LIBCPP_HAS_NO_ATOMIC_HEADER)
# include <stdatomic.h>
#endif
#include <stdbool.h>
#include <stddef.h>
#include <stdexcept>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <stop_token>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <streambuf>
#endif
#include <string.h>
#include <string>
#include <string_view>
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <strstream>
#endif
#if !defined(_LIBCPP_HAS_NO_LOCALIZATION)
# include <syncstream>
#endif
#include <system_error>
#include <tgmath.h>
#include <thread>
#include <tuple>
#include <type_traits>
#include <typeindex>
#include <typeinfo>
#include <uchar.h>
#include <unordered_map>
#include <unordered_set>
#include <utility>
#include <valarray>
#include <variant>
#include <vector>
#include <version>
#include <wchar.h>
#include <wctype.h>

4192
libcxx/include/module.modulemap
View File

File diff suppressed because it is too large Load Diff

14
libcxx/test/libcxx/clang_modules_include.gen.py
View File

@@ -37,13 +37,17 @@ for header in public_headers:
// TODO: Investigate this failure
// UNSUPPORTED: LIBCXX-FREEBSD-FIXME
// TODO: Investigate why this doesn't work on Picolibc once the locale base API is refactored
// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
{lit_header_restrictions.get(header, '')}
#include <{header}>
""")
print(f"""\
//--- __std_clang_module.compile.pass.mm
print(
f"""\
//--- import_std.compile.pass.mm
// RUN: %{{cxx}} %s %{{flags}} %{{compile_flags}} -fmodules -fcxx-modules -fmodules-cache-path=%t -fsyntax-only
// REQUIRES: clang-modules-build
@@ -61,6 +65,10 @@ print(f"""\
// TODO: Investigate this failure
// UNSUPPORTED: LIBCXX-FREEBSD-FIXME
// TODO: Investigate why this doesn't work on Picolibc once the locale base API is refactored
// UNSUPPORTED: LIBCXX-PICOLIBC-FIXME
@import std;
""")
"""
)

23
libcxx/test/std/experimental/utilities/utility/utility.synop/includes.pass.cpp
View File

@@ -1,23 +0,0 @@
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
// <experimental/utility>
#include <experimental/utility>
#include "test_macros.h"
#ifndef _LIBCPP_UTILITY
# error "<experimental/utility> must include <utility>"
#endif
int main(int, char**)
{
return 0;
}

5
libcxx/utils/CMakeLists.txt
View File

@@ -2,10 +2,6 @@ add_custom_target(libcxx-generate-feature-test-macros
COMMAND "${Python3_EXECUTABLE}" "${LIBCXX_SOURCE_DIR}/utils/generate_feature_test_macro_components.py"
COMMENT "Generate the <version> header and tests for feature test macros.")
add_custom_target(libcxx-generate-std-clang-module-header
COMMAND "${Python3_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/generate_std_clang_module_header.py"
COMMENT "Generate the <__std_clang_module> header")
add_custom_target(libcxx-generate-std-cppm-in-file
COMMAND
"${Python3_EXECUTABLE}"
@@ -57,7 +53,6 @@ add_custom_target(libcxx-indic-conjunct-break-table
add_custom_target(libcxx-generate-files
DEPENDS libcxx-generate-feature-test-macros
libcxx-generate-std-clang-module-header
libcxx-generate-std-cppm-in-file
libcxx-generate-std-compat-cppm-in-file
libcxx-generate-extended-grapheme-cluster-tables

63
libcxx/utils/generate_std_clang_module_header.py
View File

@@ -1,63 +0,0 @@
# ===----------------------------------------------------------------------===##
#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https://llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#
# ===----------------------------------------------------------------------===##
import os.path
import libcxx.header_information
header_restrictions = libcxx.header_information.header_restrictions
libcxx_include_directory = os.path.join(
os.path.dirname(os.path.dirname(os.path.realpath(__file__))), "include"
)
with open(
os.path.join(libcxx_include_directory, "__std_clang_module"), "w"
) as std_clang_module_header:
std_clang_module_header.write(
"""\
// -*- C++ -*-
//===----------------------------------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
// WARNING, this entire header is generated by
// utils/generate_std_clang_module_header.py
// DO NOT MODIFY!
// This header should not be directly included, it's exclusively to import all
// of the libc++ public clang modules for the `std` clang module to export. In
// other words, it's to facilitate `@import std;` in Objective-C++ and `import std`
// in Swift to expose all of the libc++ interfaces. This is generally not
// recommended, however there are some clients that need to import all of libc++
// without knowing what "all" is.
#if !__building_module(std)
# error "Do not include this header directly, include individual headers instead"
#endif
#include <__config>
#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
# pragma GCC system_header
#endif
"""
)
# Include the angle brackets in sorting so that <a.h> sorts before <a>
# like check-format wants.
for include, header in sorted([(f"<{header}>", header) for header in libcxx.header_information.public_headers]):
header_restriction = header_restrictions.get(header)
if header_restriction:
std_clang_module_header.write(f"#if {header_restriction}\n")
std_clang_module_header.write(f"# include {include}\n")
std_clang_module_header.write(f"#endif\n")
else:
std_clang_module_header.write(f"#include {include}\n")