Files
clice/scripts/upload-llvm.py
ykiko 592b37417e feat: cross-compile & upgrade LLVM to 21.1.8 (#390)
## Summary

This PR adds cross-compilation support for three new target platforms,
upgrades LLVM to 21.1.8, and overhauls the CI pipelines around
cross-builds and testing.

## Cross-compilation

New target triples accepted via `-DCLICE_TARGET_TRIPLE=...`:

| Target triple | Host | Output |
|---|---|---|
| `x86_64-apple-darwin` | macos-15 (arm64) | macOS x64 |
| `aarch64-linux-gnu` | ubuntu-24.04 (x64) | Linux arm64 |
| `aarch64-pc-windows-msvc` | windows-2025 (x64) | Windows arm64 |

- `cmake/toolchain.cmake` — maps `CLICE_TARGET_TRIPLE` to
`CMAKE_SYSTEM_NAME`/`CMAKE_SYSTEM_PROCESSOR`/compiler `--target`; picks
up conda aarch64 sysroot when cross-compiling Linux.
- `cmake/llvm.cmake` — forwards target platform/arch to `setup-llvm.py`
so the right prebuilt LLVM is downloaded for the target.
- `CMakeLists.txt` — uses a host-side `flatc` from `PATH` under
`CMAKE_CROSSCOMPILING` instead of the in-tree target build.
- `pixi.toml`:
  - Adds `osx-64`, `linux-aarch64`, `win-arm64` platforms.
- New environments: `cross-macos-x64`, `cross-linux-aarch64` (adds
`gcc_linux-aarch64` + `sysroot_linux-aarch64`), `cross-windows-arm64`.
- New lightweight `test-run` env used on native ARM/x64 runners to
execute cross-built artifacts (pulls in upstream clang+lld on macOS so
tests don't fall back to Apple clang).
- `scripts/activate_cross_linux.sh` — exports `CONDA_PREFIX`-relative
paths for the aarch64 toolchain.
- `scripts/build-llvm.py` — `--target-triple` support and a
`build_native_tools()` helper that produces host `llvm-tblgen` /
`clang-tblgen` needed when cross-compiling LLVM itself.

## LLVM upgrade 21.1.4 → 21.1.8

- `cmake/package.cmake` bumps `setup_llvm("21.1.8")`.
- `config/llvm-manifest.json` regenerated with 6 new cross-compiled
entries and a new `arch` field on every entry so lookup is `(version,
platform, arch, lto, build_type)`.
- `scripts/setup-llvm.py` — honours the new `arch` field when resolving
artifacts.
- `scripts/update-llvm-version.py` (new) — single-call version bump
across `package.cmake` + manifest.
- `scripts/validate-llvm-components.py` (new) — scans the LLVM source
tree for library targets and diffs them against
`scripts/llvm-components.json` to catch stale/misspelled component names
before a build.
- `scripts/llvm-components.json` (new) — explicit allow-list of required
LLVM/Clang library targets used by `build-llvm.py`.

## CI changes

- `.github/workflows/build-llvm.yml`:
- Adds `workflow_dispatch` with `llvm_version`, `skip_upload`, `skip_pr`
inputs.
- Matrix extended with the 6 cross-compile entries (2 per new platform:
RelWithDebInfo ± LTO).
- `build clice` / test / prune steps gated on `!matrix.target_triple`
for cross-builds; cross-built LTO entries apply the native prune
manifest (arch-independent).
  - Cross-compiled binary architecture is verified with `file(1)`.
- New `upload` job triggered by `workflow_dispatch` pushes artifacts to
`clice-io/clice-llvm` and hands the manifest off to the next job.
- `.github/workflows/test-cmake.yml`:
- Build matrix gains three `build_only: true` cross entries that upload
`bin/` + `lib/` artifacts.
- New `test-cross` job runs on native `macos-15-intel`,
`ubuntu-24.04-arm`, `windows-11-arm` runners, downloads the cross-built
artifacts, and runs unit / integration / smoke tests under the
`test-run` pixi env.
- Cache keys now include `target_triple` so native and cross builds
don't collide.
- `.github/workflows/publish-clice.yml`:
- Three additional release artifacts for the new targets
(`clice-x86_64-macos-darwin`, `clice-aarch64-linux-gnu`,
`clice-aarch64-windows-msvc`), each with a matching `-symbol` archive.

## Compatibility

- All existing native builds and tests are preserved; cross entries are
additive.
- `Debug` + ASAN remains disabled on Windows (`llvm_mode == Debug && os
== windows-*` no longer appends `-asan`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-19 00:17:39 +08:00

130 lines
3.5 KiB
Python

#!/usr/bin/env python3
import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path
def sha256sum(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def parse_platform(name: str) -> str:
lowered = name.lower()
if "windows" in lowered:
return "windows"
if "linux" in lowered:
return "linux"
if "macos" in lowered:
return "macosx"
raise ValueError(f"Unable to determine platform from filename: {name}")
def parse_arch(name: str) -> str:
lowered = name.lower()
if lowered.startswith("aarch64-") or lowered.startswith("arm64-"):
return "arm64"
if lowered.startswith("x64-") or lowered.startswith("x86_64-"):
return "x64"
raise ValueError(f"Unable to determine arch from filename: {name}")
def parse_build_type(name: str) -> str:
lowered = name.lower()
if "debug" in lowered:
return "Debug"
return "RelWithDebInfo"
def build_metadata_entry(path: Path, version: str) -> dict:
filename = path.name
return {
"version": version,
"filename": filename,
"sha256": sha256sum(path),
"lto": "-lto" in filename.lower(),
"asan": "-asan" in filename.lower(),
"platform": parse_platform(filename),
"arch": parse_arch(filename),
"build_type": parse_build_type(filename),
}
def main() -> None:
if len(sys.argv) != 4:
print(
"Usage: upload-llvm.py <tag> <target_repo> <workflow_id>", file=sys.stderr
)
sys.exit(1)
tag, target_repo, workflow_id = sys.argv[1:]
artifacts_dir = Path("artifacts")
if not artifacts_dir.is_dir():
print(f"Artifacts directory not found: {artifacts_dir}", file=sys.stderr)
sys.exit(1)
artifact_files = sorted(
p
for p in artifacts_dir.rglob("*")
if p.is_file() and p.suffix.lower() != ".json"
)
if not artifact_files:
print("No artifacts found to upload.", file=sys.stderr)
sys.exit(1)
version_without_prefix = tag.lstrip("vV")
metadata = [
build_metadata_entry(path, version_without_prefix) for path in artifact_files
]
json_path = artifacts_dir / "llvm-manifest.json"
with json_path.open("w", encoding="utf-8") as handle:
json.dump(metadata, handle, indent=2)
handle.write("\n")
assets = [str(path) for path in artifact_files]
assets.append(str(json_path))
env = os.environ.copy()
print(f"Checking for existing release {tag} in {target_repo}...")
view_result = subprocess.run(
["gh", "release", "view", tag, "--repo", target_repo],
env=env,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
if view_result.returncode == 0:
print(f"Deleting existing release {tag} in {target_repo}...")
subprocess.run(
["gh", "release", "delete", tag, "--repo", target_repo, "-y"],
env=env,
check=True,
)
print(f"Creating release {tag} in {target_repo} with {len(assets)} assets...")
# fmt: off
args = [
"gh", "release", "create", tag, *assets,
"--repo", target_repo,
"--title", tag,
"--notes", f"Artifacts build from workflow run https://github.com/clice-io/clice/actions/runs/{workflow_id}",
"--latest",
]
# fmt: on
subprocess.run(args, env=env, check=True)
if __name__ == "__main__":
main()