caio/clice
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: cross-compile & upgrade LLVM to 21.1.8 (#390)

Browse Source 
## Summary

This PR adds cross-compilation support for three new target platforms,
upgrades LLVM to 21.1.8, and overhauls the CI pipelines around
cross-builds and testing.

## Cross-compilation

New target triples accepted via `-DCLICE_TARGET_TRIPLE=...`:

| Target triple | Host | Output |
|---|---|---|
| `x86_64-apple-darwin` | macos-15 (arm64) | macOS x64 |
| `aarch64-linux-gnu` | ubuntu-24.04 (x64) | Linux arm64 |
| `aarch64-pc-windows-msvc` | windows-2025 (x64) | Windows arm64 |

- `cmake/toolchain.cmake` — maps `CLICE_TARGET_TRIPLE` to
`CMAKE_SYSTEM_NAME`/`CMAKE_SYSTEM_PROCESSOR`/compiler `--target`; picks
up conda aarch64 sysroot when cross-compiling Linux.
- `cmake/llvm.cmake` — forwards target platform/arch to `setup-llvm.py`
so the right prebuilt LLVM is downloaded for the target.
- `CMakeLists.txt` — uses a host-side `flatc` from `PATH` under
`CMAKE_CROSSCOMPILING` instead of the in-tree target build.
- `pixi.toml`:
  - Adds `osx-64`, `linux-aarch64`, `win-arm64` platforms.
- New environments: `cross-macos-x64`, `cross-linux-aarch64` (adds
`gcc_linux-aarch64` + `sysroot_linux-aarch64`), `cross-windows-arm64`.
- New lightweight `test-run` env used on native ARM/x64 runners to
execute cross-built artifacts (pulls in upstream clang+lld on macOS so
tests don't fall back to Apple clang).
- `scripts/activate_cross_linux.sh` — exports `CONDA_PREFIX`-relative
paths for the aarch64 toolchain.
- `scripts/build-llvm.py` — `--target-triple` support and a
`build_native_tools()` helper that produces host `llvm-tblgen` /
`clang-tblgen` needed when cross-compiling LLVM itself.

## LLVM upgrade 21.1.4 → 21.1.8

- `cmake/package.cmake` bumps `setup_llvm("21.1.8")`.
- `config/llvm-manifest.json` regenerated with 6 new cross-compiled
entries and a new `arch` field on every entry so lookup is `(version,
platform, arch, lto, build_type)`.
- `scripts/setup-llvm.py` — honours the new `arch` field when resolving
artifacts.
- `scripts/update-llvm-version.py` (new) — single-call version bump
across `package.cmake` + manifest.
- `scripts/validate-llvm-components.py` (new) — scans the LLVM source
tree for library targets and diffs them against
`scripts/llvm-components.json` to catch stale/misspelled component names
before a build.
- `scripts/llvm-components.json` (new) — explicit allow-list of required
LLVM/Clang library targets used by `build-llvm.py`.

## CI changes

- `.github/workflows/build-llvm.yml`:
- Adds `workflow_dispatch` with `llvm_version`, `skip_upload`, `skip_pr`
inputs.
- Matrix extended with the 6 cross-compile entries (2 per new platform:
RelWithDebInfo ± LTO).
- `build clice` / test / prune steps gated on `!matrix.target_triple`
for cross-builds; cross-built LTO entries apply the native prune
manifest (arch-independent).
  - Cross-compiled binary architecture is verified with `file(1)`.
- New `upload` job triggered by `workflow_dispatch` pushes artifacts to
`clice-io/clice-llvm` and hands the manifest off to the next job.
- `.github/workflows/test-cmake.yml`:
- Build matrix gains three `build_only: true` cross entries that upload
`bin/` + `lib/` artifacts.
- New `test-cross` job runs on native `macos-15-intel`,
`ubuntu-24.04-arm`, `windows-11-arm` runners, downloads the cross-built
artifacts, and runs unit / integration / smoke tests under the
`test-run` pixi env.
- Cache keys now include `target_triple` so native and cross builds
don't collide.
- `.github/workflows/publish-clice.yml`:
- Three additional release artifacts for the new targets
(`clice-x86_64-macos-darwin`, `clice-aarch64-linux-gnu`,
`clice-aarch64-windows-msvc`), each with a matching `-symbol` archive.

## Compatibility

- All existing native builds and tests are preserved; cross entries are
additive.
- `Debug` + ASAN remains disabled on Windows (`llvm_mode == Debug && os
== windows-*` no longer appends `-asan`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
ykiko
2026-04-19 00:17:39 +08:00
committed by GitHub
parent 418e190fa0
commit 592b37417e
22 changed files with 6741 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
scripts/activate_cross_linux.sh Normal file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# Clear conda cross-gcc flags so host x86_64 paths don't leak into the
# aarch64 build. conda's gcc_linux-aarch64 activation sets
# CFLAGS/CXXFLAGS/CPPFLAGS/LDFLAGS with -isystem/-L pointing at $CONDA_PREFIX
# (x86_64 host paths). LIBRARY_PATH from ld_impl_linux-64 likewise points at
# host libs. Empty-string export reliably overrides conda-installed values
# regardless of whether pixi sources or calls this script.
export CFLAGS=
export CXXFLAGS=
export CPPFLAGS=
export LDFLAGS=
export LIBRARY_PATH=

8
scripts/activate_cross_macos.sh Normal file
View File

@@ -0,0 +1,8 @@
#!/bin/sh
# Clear conda host flags so arm64 host paths don't leak into the x86_64-macos
# cross build. See scripts/activate_cross_linux.sh for rationale.
export CFLAGS=
export CXXFLAGS=
export CPPFLAGS=
export LDFLAGS=
export LIBRARY_PATH=

8
scripts/activate_cross_windows.bat Normal file
View File

@@ -0,0 +1,8 @@
@echo off
REM Clear conda host flags so host x64 paths don't leak into the aarch64-windows
REM cross build. See scripts/activate_cross_linux.sh for rationale.
set "CFLAGS="
set "CXXFLAGS="
set "CPPFLAGS="
set "LDFLAGS="
set "LIBRARY_PATH="

218
scripts/build-llvm.py
View File

@@ -4,6 +4,7 @@ import subprocess
import shutil
import argparse
import os
import json
from pathlib import Path
@@ -22,6 +23,66 @@ def normalize_mode(value: str) -> str:
)
def build_native_tools(project_root: Path, build_dir: Path) -> Path:
"""Build native host tablegen tools for cross-compilation.
When cross-compiling LLVM, build tools like llvm-tblgen must run on the
host but would otherwise be compiled for the target architecture. This
function performs a minimal native build and returns the bin directory
containing host-runnable executables.
"""
native_dir = build_dir.parent / f"{build_dir.name}-native-tools"
native_dir.mkdir(exist_ok=True)
source_dir = project_root / "llvm"
cmake_args = [
"-G",
"Ninja",
"-DCMAKE_BUILD_TYPE=Release",
"-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra",
"-DLLVM_TARGETS_TO_BUILD=Native",
"-DLLVM_DISABLE_ASSEMBLY_FILES=ON",
"-DCMAKE_C_FLAGS=-w",
"-DCMAKE_CXX_FLAGS=-w",
]
if sys.platform == "win32":
cmake_args += [
"-DCMAKE_C_COMPILER=clang-cl",
"-DCMAKE_CXX_COMPILER=clang-cl",
]
else:
cmake_args += [
"-DCMAKE_C_COMPILER=clang",
"-DCMAKE_CXX_COMPILER=clang++",
]
print(f"\nConfiguring native host tools in {native_dir}...")
subprocess.check_call(
["cmake", "-S", str(source_dir), "-B", str(native_dir)] + cmake_args
)
required_tools = ["llvm-tblgen", "llvm-min-tblgen", "clang-tblgen"]
optional_tools = ["clang-tidy-confusable-chars-gen"]
for tool in required_tools:
print(f"Building native {tool}...")
subprocess.check_call(["cmake", "--build", str(native_dir), "--target", tool])
for tool in optional_tools:
try:
print(f"Building native {tool} (optional)...")
subprocess.check_call(
["cmake", "--build", str(native_dir), "--target", tool]
)
except subprocess.CalledProcessError:
print(f" {tool} not available, skipping.")
bin_dir = native_dir / "bin"
print(f"Native host tools ready in {bin_dir}")
return bin_dir
def main():
parser = argparse.ArgumentParser(
description="Build LLVM with specific configurations."
@@ -48,6 +109,10 @@ def main():
"--build-dir",
help="Custom build directory (relative to project root or absolute)",
)
parser.add_argument(
"--target-triple",
help="Cross-compilation target triple (e.g. x86_64-apple-darwin, aarch64-linux-gnu, aarch64-pc-windows-msvc)",
)
args = parser.parse_args()
@@ -85,118 +150,46 @@ def main():
print("--- Configuration ---")
print(f"Mode: {args.mode}")
print(f"LTO: {args.lto}")
print(f"Target Triple: {args.target_triple or '(native)'}")
print(f"Root: {project_root}")
print(f"Build Dir: {build_dir}")
print(f"Install Prefix: {install_prefix}")
print(f"Toolchain: {toolchain_file}")
print("---------------------")
llvm_distribution_components = [
"LLVMDemangle",
"LLVMSupport",
"LLVMCore",
"LLVMOption",
"LLVMBinaryFormat",
"LLVMMC",
"LLVMMCParser",
"LLVMObject",
"LLVMProfileData",
"LLVMBitReader",
"LLVMBitstreamReader",
"LLVMRemarks",
"LLVMObjectYAML",
"LLVMAggressiveInstCombine",
"LLVMInstCombine",
"LLVMIRReader",
"LLVMTextAPI",
"LLVMSymbolize",
"LLVMDebugInfoDWARF",
"LLVMDebugInfoDWARFLowLevel",
"LLVMDebugInfoCodeView",
"LLVMDebugInfoGSYM",
"LLVMDebugInfoPDB",
"LLVMDebugInfoBTF",
"LLVMDebugInfoMSF",
"LLVMAsmParser",
"LLVMTargetParser",
"LLVMTransformUtils",
"LLVMAnalysis",
"LLVMScalarOpts",
"LLVMFrontendHLSL",
"LLVMFrontendOpenMP",
"LLVMFrontendOffloading",
"LLVMFrontendAtomic",
"LLVMFrontendDirective",
"LLVMWindowsDriver",
"clangIndex",
"clangAPINotes",
"clangAST",
"clangASTMatchers",
"clangBasic",
"clangDriver",
"clangFormat",
"clangFrontend",
"clangLex",
"clangParse",
"clangSema",
"clangSerialization",
"clangRewrite",
"clangAnalysis",
"clangEdit",
"clangSupport",
"clangStaticAnalyzerCore",
"clangStaticAnalyzerFrontend",
"clangTidy",
"clangTidyUtils",
"clangTidyAndroidModule",
"clangTidyAbseilModule",
"clangTidyAlteraModule",
"clangTidyBoostModule",
"clangTidyBugproneModule",
"clangTidyCERTModule",
"clangTidyConcurrencyModule",
"clangTidyCppCoreGuidelinesModule",
"clangTidyDarwinModule",
"clangTidyFuchsiaModule",
"clangTidyGoogleModule",
"clangTidyHICPPModule",
"clangTidyLinuxKernelModule",
"clangTidyLLVMModule",
"clangTidyLLVMLibcModule",
"clangTidyMiscModule",
"clangTidyModernizeModule",
"clangTidyObjCModule",
"clangTidyOpenMPModule",
"clangTidyPerformanceModule",
"clangTidyPortabilityModule",
"clangTidyReadabilityModule",
"clangTidyZirconModule",
"clangTooling",
"clangToolingCore",
"clangToolingInclusions",
"clangToolingInclusionsStdlib",
"clangToolingSyntax",
"clangToolingRefactoring",
"clangTransformer",
"clangCrossTU",
"clangAnalysisFlowSensitive",
"clangAnalysisFlowSensitiveModels",
"clangStaticAnalyzerCheckers",
"clangIncludeCleaner",
"llvm-headers",
"clang-headers",
"clang-tidy-headers",
"clang-resource-headers",
]
components_path = Path(__file__).resolve().parent / "llvm-components.json"
with components_path.open() as f:
llvm_distribution_components = json.load(f)["components"]
components_joined = ";".join(llvm_distribution_components)
cmake_args = [
"-G",
"Ninja",
f"-DCMAKE_TOOLCHAIN_FILE={toolchain_file.as_posix()}",
f"-DCMAKE_INSTALL_PREFIX={install_prefix}",
"-DCMAKE_C_FLAGS=-w",
"-DCMAKE_CXX_FLAGS=-w",
]
if sys.platform == "win32":
# Use clang-cl (MSVC driver) on Windows so that LLVM's CMake
# generates correct MSVC-style linker flags for LTO, etc.
c_flags = "-w"
if args.target_triple:
c_flags += f" --target={args.target_triple}"
cmake_args += [
"-DCMAKE_C_COMPILER=clang-cl",
"-DCMAKE_CXX_COMPILER=clang-cl",
f"-DCMAKE_C_FLAGS={c_flags}",
f"-DCMAKE_CXX_FLAGS={c_flags}",
"-DLLVM_USE_LINKER=lld-link",
]
else:
cmake_args += [
f"-DCMAKE_TOOLCHAIN_FILE={toolchain_file.as_posix()}",
"-DCMAKE_C_FLAGS=-w",
"-DCMAKE_CXX_FLAGS=-w",
"-DLLVM_USE_LINKER=lld",
]
cmake_args += [
"-DLLVM_ENABLE_ZLIB=OFF",
"-DLLVM_ENABLE_ZSTD=OFF",
"-DLLVM_ENABLE_LIBXML2=OFF",
@@ -231,7 +224,6 @@ def main():
"-DCMAKE_JOB_POOL_LINK=console",
"-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra",
"-DLLVM_TARGETS_TO_BUILD=all",
"-DLLVM_USE_LINKER=lld",
"-DLLVM_DISABLE_ASSEMBLY_FILES=ON",
# Distribution
f"-DLLVM_DISTRIBUTION_COMPONENTS={components_joined}",
@@ -256,8 +248,10 @@ def main():
is_shared = "OFF"
if args.mode == "Debug":
cmake_args.append("-DCMAKE_BUILD_TYPE=Debug")
cmake_args.append("-DLLVM_USE_SANITIZER=Address")
is_shared = "ON"
# ASAN is incompatible with -MDd on Windows (clang-cl), skip it there.
if sys.platform != "win32":
cmake_args.append("-DLLVM_USE_SANITIZER=Address")
is_shared = "ON"
elif args.mode == "Release":
cmake_args.append("-DCMAKE_BUILD_TYPE=Release")
elif args.mode == "RelWithDebInfo":
@@ -272,6 +266,24 @@ def main():
else:
cmake_args.append("-DLLVM_ENABLE_LTO=OFF")
if args.target_triple:
cmake_args.append(f"-DCLICE_TARGET_TRIPLE={args.target_triple}")
cmake_args.append(f"-DLLVM_HOST_TRIPLE={args.target_triple}")
# When cross-compiling, clear conda's host-platform flags so they
# don't leak into the target build (e.g. -L pointing to x86_64 libs).
# This must happen before the native-tools build too so we don't
# contaminate the native configure with target-arch link flags.
for var in ["LIBRARY_PATH", "LDFLAGS", "CFLAGS", "CXXFLAGS", "CPPFLAGS"]:
os.environ.pop(var, None)
# Cross-compilation needs native host tools (tablegen, etc.) that can
# run on the build machine. macOS handles this transparently via
# Rosetta 2, but Linux and Windows require a separate native build.
if sys.platform != "darwin":
native_bin_dir = build_native_tools(project_root, build_dir)
cmake_args.append(f"-DLLVM_NATIVE_TOOL_DIR={native_bin_dir}")
build_dir.mkdir(exist_ok=True)
print(f"\nConfiguring in {build_dir}...")

99
scripts/llvm-components.json Normal file
View File

@@ -0,0 +1,99 @@
{
"components": [
"LLVMDemangle",
"LLVMSupport",
"LLVMCore",
"LLVMOption",
"LLVMBinaryFormat",
"LLVMMC",
"LLVMMCParser",
"LLVMObject",
"LLVMProfileData",
"LLVMBitReader",
"LLVMBitstreamReader",
"LLVMRemarks",
"LLVMObjectYAML",
"LLVMAggressiveInstCombine",
"LLVMInstCombine",
"LLVMIRReader",
"LLVMTextAPI",
"LLVMSymbolize",
"LLVMDebugInfoDWARF",
"LLVMDebugInfoDWARFLowLevel",
"LLVMDebugInfoCodeView",
"LLVMDebugInfoGSYM",
"LLVMDebugInfoPDB",
"LLVMDebugInfoBTF",
"LLVMDebugInfoMSF",
"LLVMAsmParser",
"LLVMTargetParser",
"LLVMTransformUtils",
"LLVMAnalysis",
"LLVMScalarOpts",
"LLVMFrontendHLSL",
"LLVMFrontendOpenMP",
"LLVMFrontendOffloading",
"LLVMFrontendAtomic",
"LLVMFrontendDirective",
"LLVMWindowsDriver",
"clangIndex",
"clangAPINotes",
"clangAST",
"clangASTMatchers",
"clangBasic",
"clangDriver",
"clangFormat",
"clangFrontend",
"clangLex",
"clangParse",
"clangSema",
"clangSerialization",
"clangRewrite",
"clangAnalysis",
"clangEdit",
"clangSupport",
"clangStaticAnalyzerCore",
"clangStaticAnalyzerFrontend",
"clangTidy",
"clangTidyUtils",
"clangTidyAndroidModule",
"clangTidyAbseilModule",
"clangTidyAlteraModule",
"clangTidyBoostModule",
"clangTidyBugproneModule",
"clangTidyCERTModule",
"clangTidyConcurrencyModule",
"clangTidyCppCoreGuidelinesModule",
"clangTidyDarwinModule",
"clangTidyFuchsiaModule",
"clangTidyGoogleModule",
"clangTidyHICPPModule",
"clangTidyLinuxKernelModule",
"clangTidyLLVMModule",
"clangTidyLLVMLibcModule",
"clangTidyMiscModule",
"clangTidyModernizeModule",
"clangTidyObjCModule",
"clangTidyOpenMPModule",
"clangTidyPerformanceModule",
"clangTidyPortabilityModule",
"clangTidyReadabilityModule",
"clangTidyZirconModule",
"clangTooling",
"clangToolingCore",
"clangToolingInclusions",
"clangToolingInclusionsStdlib",
"clangToolingSyntax",
"clangToolingRefactoring",
"clangTransformer",
"clangCrossTU",
"clangAnalysisFlowSensitive",
"clangAnalysisFlowSensitiveModels",
"clangStaticAnalyzerCheckers",
"clangIncludeCleaner",
"llvm-headers",
"clang-headers",
"clang-tidy-headers",
"clang-resource-headers"
]
}

62
scripts/setup-llvm.py
View File

@@ -40,23 +40,52 @@ def detect_platform() -> str:
raise RuntimeError(f"Unsupported platform: {plat}")
def detect_arch() -> str:
import platform
machine = platform.machine().lower()
if machine in ("x86_64", "amd64"):
return "x64"
if machine in ("aarch64", "arm64"):
return "arm64"
raise RuntimeError(f"Unsupported architecture: {machine}")
def pick_artifact(
manifest: list[dict], version: str, build_type: str, is_lto: bool, platform: str
manifest: list[dict],
version: str,
build_type: str,
is_lto: bool,
platform: str,
arch: str,
) -> dict:
base_version = version.split("+", 1)[0]
saw_missing_arch = False
for entry in manifest:
if entry.get("version") != version:
continue
if entry.get("platform") != platform.lower():
continue
entry_arch = entry.get("arch")
if entry_arch is None:
saw_missing_arch = True
continue
if entry_arch != arch:
continue
if entry.get("build_type") != build_type:
continue
if bool(entry.get("lto")) != is_lto:
continue
return entry
if saw_missing_arch:
raise RuntimeError(
f"Manifest contains entries without an 'arch' field for version={base_version}, "
f"platform={platform}. The manifest format changed to require explicit "
f"architectures; regenerate it via scripts/update-llvm-version.py."
)
raise RuntimeError(
f"No matching LLVM artifact in manifest for version={base_version}, platform={platform}, "
f"build_type={build_type}, lto={is_lto}"
f"arch={arch}, build_type={build_type}, lto={is_lto}"
)
@@ -264,6 +293,14 @@ def main() -> None:
parser.add_argument("--install-path")
parser.add_argument("--enable-lto", action="store_true")
parser.add_argument("--offline", action="store_true")
parser.add_argument(
"--target-platform",
help="Override platform for cross-compilation (e.g. macosx, linux, windows)",
)
parser.add_argument(
"--target-arch",
help="Override architecture for cross-compilation (e.g. x64, arm64)",
)
parser.add_argument("--output", required=True)
args = parser.parse_args()
@@ -275,8 +312,11 @@ def main() -> None:
)
token = os.environ.get("GH_TOKEN") or os.environ.get("GITHUB_TOKEN")
build_type = args.build_type
platform_name = detect_platform()
log(f"Platform detected: {platform_name}, normalized build type: {build_type}")
platform_name = args.target_platform if args.target_platform else detect_platform()
arch_name = args.target_arch if args.target_arch else detect_arch()
log(
f"Platform: {platform_name}, arch: {arch_name}, normalized build type: {build_type}"
)
manifest = read_manifest(Path(args.manifest))
binary_dir = Path(args.binary_dir).resolve()
@@ -304,7 +344,12 @@ def main() -> None:
if install_path is None:
needs_install = True
artifact = pick_artifact(
manifest, args.version, build_type, args.enable_lto, platform_name
manifest,
args.version,
build_type,
args.enable_lto,
platform_name,
arch_name,
)
log(f"Selected artifact: {artifact.get('filename')} for download")
filename = artifact["filename"]
@@ -317,7 +362,12 @@ def main() -> None:
install_path = install_root
elif needs_install:
artifact = pick_artifact(
manifest, args.version, build_type, args.enable_lto, platform_name
manifest,
args.version,
build_type,
args.enable_lto,
platform_name,
arch_name,
)
log(f"Selected artifact: {artifact.get('filename')} for download")
filename = artifact["filename"]

162
scripts/update-llvm-version.py Executable file
View File

@@ -0,0 +1,162 @@
#!/usr/bin/env python3
import argparse
import json
import re
import sys
from pathlib import Path
def copy_manifest(src: Path, dest: Path) -> None:
text = src.read_text(encoding="utf-8")
try:
data = json.loads(text)
except json.JSONDecodeError as err:
print(f"Error: {src} is not valid JSON: {err}", file=sys.stderr)
sys.exit(1)
if not isinstance(data, list) or len(data) == 0:
print(f"Error: {src} must be a non-empty JSON array", file=sys.stderr)
sys.exit(1)
dest.parent.mkdir(parents=True, exist_ok=True)
with dest.open("w", encoding="utf-8") as handle:
json.dump(data, handle, indent=2)
handle.write("\n")
print(f"Copied manifest: {src} -> {dest} ({len(data)} entries)")
def update_package_cmake(path: Path, version: str) -> None:
text = path.read_text(encoding="utf-8")
pattern = r'setup_llvm\("[^"]*"\)'
matches = re.findall(pattern, text)
if len(matches) == 0:
print(f"Error: no setup_llvm(...) call found in {path}", file=sys.stderr)
sys.exit(1)
if len(matches) > 1:
print(
f"Error: expected exactly 1 setup_llvm(...) call in {path}, "
f"found {len(matches)}",
file=sys.stderr,
)
sys.exit(1)
old_call = matches[0]
new_call = f'setup_llvm("{version}")'
if old_call == new_call:
print(f"Version in {path} is already {version}, no change needed")
return
updated = text.replace(old_call, new_call)
path.write_text(updated, encoding="utf-8")
print(f"Updated {path}: {old_call} -> {new_call}")
def check_package_cmake(path: Path) -> None:
"""Verify package.cmake has exactly one setup_llvm(...) call that the
update script can rewrite. Used by CI to catch drift before the next bump."""
text = path.read_text(encoding="utf-8")
matches = re.findall(r'setup_llvm\("[^"]*"\)', text)
if len(matches) == 0:
print(f"Error: no setup_llvm(...) call found in {path}", file=sys.stderr)
sys.exit(1)
if len(matches) > 1:
print(
f"Error: expected exactly 1 setup_llvm(...) call in {path}, "
f"found {len(matches)}: {matches}",
file=sys.stderr,
)
sys.exit(1)
print(f"OK: {path} has a single setup_llvm(...) call: {matches[0]}")
def check_manifest(path: Path) -> None:
"""Verify the manifest is a well-formed non-empty array with required fields."""
try:
data = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as err:
print(f"Error: {path} is not valid JSON: {err}", file=sys.stderr)
sys.exit(1)
if not isinstance(data, list) or len(data) == 0:
print(f"Error: {path} must be a non-empty JSON array", file=sys.stderr)
sys.exit(1)
required = ("version", "platform", "arch", "build_type", "filename", "sha256")
for idx, entry in enumerate(data):
missing = [k for k in required if k not in entry]
if missing:
print(
f"Error: {path} entry {idx} is missing fields: {missing}",
file=sys.stderr,
)
sys.exit(1)
print(f"OK: {path} has {len(data)} well-formed entries")
def main() -> None:
parser = argparse.ArgumentParser(
description="Update LLVM version references in the clice project."
)
parser.add_argument(
"--check",
action="store_true",
help="Validate existing state without modifying files (for CI drift checks)",
)
parser.add_argument(
"--version",
help="New LLVM version string (e.g. 21.2.0); required unless --check",
)
parser.add_argument(
"--manifest-src",
help="Path to the source llvm-manifest.json; required unless --check",
)
parser.add_argument(
"--manifest-dest",
required=True,
help="Path to destination manifest (e.g. config/llvm-manifest.json)",
)
parser.add_argument(
"--package-cmake",
required=True,
help="Path to cmake/package.cmake",
)
args = parser.parse_args()
manifest_dest = Path(args.manifest_dest)
package_cmake = Path(args.package_cmake)
if not package_cmake.is_file():
print(f"Error: package.cmake not found: {package_cmake}", file=sys.stderr)
sys.exit(1)
if args.check:
check_package_cmake(package_cmake)
check_manifest(manifest_dest)
print("Done (check mode).")
return
if not args.version or not args.manifest_src:
print(
"Error: --version and --manifest-src are required unless --check is set",
file=sys.stderr,
)
sys.exit(1)
manifest_src = Path(args.manifest_src)
if not manifest_src.is_file():
print(f"Error: manifest source not found: {manifest_src}", file=sys.stderr)
sys.exit(1)
copy_manifest(manifest_src, manifest_dest)
update_package_cmake(package_cmake, args.version)
print("Done.")
if __name__ == "__main__":
main()

10
scripts/upload-llvm.py
View File

@@ -27,6 +27,15 @@ def parse_platform(name: str) -> str:
raise ValueError(f"Unable to determine platform from filename: {name}")
def parse_arch(name: str) -> str:
lowered = name.lower()
if lowered.startswith("aarch64-") or lowered.startswith("arm64-"):
return "arm64"
if lowered.startswith("x64-") or lowered.startswith("x86_64-"):
return "x64"
raise ValueError(f"Unable to determine arch from filename: {name}")
def parse_build_type(name: str) -> str:
lowered = name.lower()
if "debug" in lowered:
@@ -43,6 +52,7 @@ def build_metadata_entry(path: Path, version: str) -> dict:
"lto": "-lto" in filename.lower(),
"asan": "-asan" in filename.lower(),
"platform": parse_platform(filename),
"arch": parse_arch(filename),
"build_type": parse_build_type(filename),
}

163
scripts/validate-llvm-components.py Executable file
View File

@@ -0,0 +1,163 @@
#!/usr/bin/env python3
"""
Validate the LLVM distribution component list against the actual LLVM source tree.
Scans the LLVM source for CMake library targets and compares them against
a components JSON file to detect stale or misspelled entries.
"""
import argparse
import difflib
import json
import re
import sys
from pathlib import Path
# CMake function calls that define library targets.
# The captured group uses [^\s)]+ to grab the target name without
# trailing parentheses or whitespace.
LLVM_LIB_PATTERNS = [
re.compile(r"add_llvm_component_library\(\s*([^\s)]+)"),
re.compile(r"add_llvm_library\(\s*([^\s)]+)"),
]
CLANG_LIB_PATTERNS = [
re.compile(r"add_clang_library\(\s*([^\s)]+)"),
]
# Header-only / custom install targets.
HEADER_PATTERNS = [
re.compile(r"add_llvm_install_targets\(\s*([^\s)]+)"),
re.compile(r"add_custom_target\(\s*([^\s)]+)"),
re.compile(r"add_library\(\s*([^\s)]+)"),
]
# Targets we recognise as header-only distribution components.
KNOWN_HEADER_TARGETS = {
"llvm-headers",
"clang-headers",
"clang-tidy-headers",
"clang-resource-headers",
}
def scan_targets(directory: Path, patterns: list[re.Pattern]) -> set[str]:
"""Recursively scan *directory* for CMakeLists.txt files and extract target names."""
targets: set[str] = set()
if not directory.is_dir():
return targets
for cmake_file in directory.rglob("CMakeLists.txt"):
text = cmake_file.read_text(errors="replace")
for pattern in patterns:
for match in pattern.finditer(text):
targets.add(match.group(1))
return targets
def scan_header_targets(llvm_src: Path) -> set[str]:
"""Scan for well-known header / custom-install targets across the tree."""
found: set[str] = set()
for cmake_file in llvm_src.rglob("CMakeLists.txt"):
text = cmake_file.read_text(errors="replace")
for pattern in HEADER_PATTERNS:
for match in pattern.finditer(text):
name = match.group(1)
if name in KNOWN_HEADER_TARGETS:
found.add(name)
return found
def collect_source_targets(llvm_src: Path) -> set[str]:
"""Return the full set of library / header targets found in the LLVM source tree."""
targets: set[str] = set()
targets |= scan_targets(llvm_src / "llvm" / "lib", LLVM_LIB_PATTERNS)
targets |= scan_targets(llvm_src / "clang" / "lib", CLANG_LIB_PATTERNS)
targets |= scan_targets(llvm_src / "clang-tools-extra", CLANG_LIB_PATTERNS)
targets |= scan_header_targets(llvm_src)
return targets
def load_components(path: Path) -> list[str]:
with path.open("r", encoding="utf-8") as handle:
data = json.load(handle)
if isinstance(data, dict):
data = data.get("components", [])
if not isinstance(data, list) or not data:
print(f"Error: no component list found in {path}", file=sys.stderr)
sys.exit(1)
return data
def main() -> None:
parser = argparse.ArgumentParser(
description="Validate LLVM distribution components against the source tree."
)
parser.add_argument(
"--llvm-src",
required=True,
help="Path to the llvm-project source root",
)
parser.add_argument(
"--components-file",
required=True,
help="Path to llvm-components.json",
)
args = parser.parse_args()
llvm_src = Path(args.llvm_src).expanduser().resolve()
components_file = Path(args.components_file).expanduser().resolve()
if not llvm_src.is_dir():
print(f"Error: LLVM source directory not found: {llvm_src}")
sys.exit(1)
if not (llvm_src / "llvm" / "CMakeLists.txt").exists():
print(f"Error: {llvm_src} does not look like an llvm-project root.")
sys.exit(1)
if not components_file.is_file():
print(f"Error: components file not found: {components_file}")
sys.exit(1)
components = load_components(components_file)
source_targets = collect_source_targets(llvm_src)
print(f"Found {len(source_targets)} targets in LLVM source tree")
print(f"Components file lists {len(components)} entries")
# Check for components that are missing from the source tree.
missing: list[tuple[str, list[str]]] = []
for name in components:
if name not in source_targets:
suggestions = difflib.get_close_matches(
name, source_targets, n=3, cutoff=0.6
)
missing.append((name, suggestions))
if missing:
print(f"\nError: {len(missing)} component(s) not found in the source tree:\n")
for name, suggestions in missing:
print(f" - {name}")
if suggestions:
print(f" Did you mean: {', '.join(suggestions)}?")
sys.exit(1)
# Warn about source targets not present in the component list.
component_set = set(components)
new_targets = sorted(source_targets - component_set - KNOWN_HEADER_TARGETS)
# Filter to targets that follow LLVM/Clang naming conventions to reduce noise.
noteworthy = [t for t in new_targets if t.startswith(("LLVM", "clang", "Clang"))]
if noteworthy:
print(
f"\nWarning: {len(noteworthy)} target(s) in source not listed in components:"
)
for name in noteworthy:
print(f" + {name}")
print("\nAll components validated successfully.")
sys.exit(0)
if __name__ == "__main__":
main()