From 5136c6d9d207b72135c92567e58e5cdbb86efc09 Mon Sep 17 00:00:00 2001 From: Marc Auberer Date: Wed, 22 Jan 2025 13:53:23 +0100 Subject: [PATCH] [Tools][Docker] Update Dockerfiles and Docker guide (#123841) Fixes #123669 - Update Dockerfiles to work with the LLVM trunk - Adapt Documentation accordingly - Fix duplicate `-c` flag --- llvm/docs/Docker.rst | 36 ++++++++----------- llvm/utils/docker/build_docker_image.sh | 16 ++++----- .../docker/{debian10 => debian12}/Dockerfile | 16 ++++----- llvm/utils/docker/example/Dockerfile | 6 ++-- llvm/utils/docker/nvidia-cuda/Dockerfile | 10 +++--- llvm/utils/docker/scripts/checkout.sh | 8 ++--- 6 files changed, 43 insertions(+), 49 deletions(-) rename llvm/utils/docker/{debian10 => debian12}/Dockerfile (79%) diff --git a/llvm/docs/Docker.rst b/llvm/docs/Docker.rst index 6b3c80da684b..5d976eddb313 100644 --- a/llvm/docs/Docker.rst +++ b/llvm/docs/Docker.rst @@ -9,7 +9,7 @@ You can find a number of sources to build docker images with LLVM components in images for their own use, or as a starting point for someone who wants to write their own Dockerfiles. -We currently provide Dockerfiles with ``debian10`` and ``nvidia-cuda`` base images. +We currently provide Dockerfiles with ``debian12`` and ``nvidia-cuda`` base images. We also provide an ``example`` image, which contains placeholders that one would need to fill out in order to produce Dockerfiles for a new docker image. @@ -72,13 +72,13 @@ checkout from git and provide a list of CMake arguments to use during when building LLVM inside docker container. Here's a very simple example of getting a docker image with clang binary, -compiled by the system compiler in the debian10 image: +compiled by the system compiler in the debian12 image: .. code-block:: bash ./llvm/utils/docker/build_docker_image.sh \ - --source debian10 \ - --docker-repository clang-debian10 --docker-tag "staging" \ + --source debian12 \ + --docker-repository clang-debian12 --docker-tag "staging" \ -p clang -i install-clang -i install-clang-resource-headers \ -- \ -DCMAKE_BUILD_TYPE=Release @@ -93,51 +93,45 @@ this command will do that: # LLVM_TARGETS_TO_BUILD=Native is to reduce stage1 compile time. # Options, starting with BOOTSTRAP_* are passed to stage2 cmake invocation. ./build_docker_image.sh \ - --source debian10 \ - --docker-repository clang-debian10 --docker-tag "staging" \ + --source debian12 \ + --docker-repository clang-debian12 --docker-tag "staging" \ -p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \ -- \ -DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \ -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \ -DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-resource-headers" -This will produce a new image ``clang-debian10:staging`` from the latest +This will produce a new image ``clang-debian12:staging`` from the latest upstream revision. After the image is built you can run bash inside a container based on your image like this: .. code-block:: bash - docker run -ti clang-debian10:staging bash + docker run -ti clang-debian12:staging bash Now you can run bash commands as you normally would: .. code-block:: bash root@80f351b51825:/# clang -v - clang version 5.0.0 (trunk 305064) + clang version 19.1.7 (trunk 524462) + Target: x86_64-unknown-linux-gnu Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /bin - Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8 - Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4 - Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9 - Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2 - Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9 - Candidate multilib: .;@m64 - Selected multilib: .;@m64 Which image should I choose? ============================ -We currently provide two images: Debian10-based and nvidia-cuda-based. They +We currently provide two images: Debian12-based and nvidia-cuda-based. They differ in the base image that they use, i.e. they have a different set of preinstalled binaries. Debian8 is very minimal, nvidia-cuda is larger, but has preinstalled CUDA libraries and allows to access a GPU, installed on your machine. If you need a minimal linux distribution with only clang and libstdc++ included, -you should try Debian10-based image. +you should try Debian12-based image. If you want to use CUDA libraries and have access to a GPU on your machine, you should choose nvidia-cuda-based image and use `nvidia-docker @@ -150,7 +144,7 @@ If you have a different use-case, you could create your own image based on ``example/`` folder. Any docker image can be built and run using only the docker binary, i.e. you can -run debian10 build on Fedora or any other Linux distribution. You don't need to +run debian12 build on Fedora or any other Linux distribution. You don't need to install CMake, compilers or any other clang dependencies. It is all handled during the build process inside Docker's isolated environment. @@ -158,12 +152,12 @@ Stable build ============ If you want a somewhat recent and somewhat stable build, use the ``branches/google/stable`` branch, i.e. the following command will produce a -Debian10-based image using the latest ``google/stable`` sources for you: +Debian12-based image using the latest ``google/stable`` sources for you: .. code-block:: bash ./llvm/utils/docker/build_docker_image.sh \ - -s debian10 --d clang-debian10 -t "staging" \ + -s debian12 --d clang-debian12 -t "staging" \ --branch branches/google/stable \ -p clang -i install-clang -i install-clang-resource-headers \ -- \ diff --git a/llvm/utils/docker/build_docker_image.sh b/llvm/utils/docker/build_docker_image.sh index f2d43934f4cc..68de54709c29 100755 --- a/llvm/utils/docker/build_docker_image.sh +++ b/llvm/utils/docker/build_docker_image.sh @@ -23,7 +23,7 @@ Available options: General: -h|--help show this help message Docker-specific: - -s|--source image source dir (i.e. debian10, nvidia-cuda, etc) + -s|--source image source dir (i.e. debian12, nvidia-cuda, etc) -d|--docker-repository docker repository for the image -t|--docker-tag docker tag for the image Checkout arguments: @@ -38,7 +38,7 @@ Available options: -p|--llvm-project Add the project to a list LLVM_ENABLE_PROJECTS, passed to CMake. Can be specified multiple times. - -c|--checksums name of a file, containing checksums of llvm checkout. + --checksums name of a file, containing checksums of llvm checkout. Script will fail if checksums of the checkout do not match. Build-specific: @@ -51,18 +51,18 @@ Required options: --source and --docker-repository, at least one All options after '--' are passed to CMake invocation. For example, running: -$ build_docker_image.sh -s debian10 -d mydocker/debian10-clang -t latest \ +$ build_docker_image.sh -s debian12 -d mydocker/debian12-clang -t latest \ -p clang -i install-clang -i install-clang-resource-headers will produce two docker images: - mydocker/debian10-clang-build:latest - an intermediate image used to compile + mydocker/debian12-clang-build:latest - an intermediate image used to compile clang. - mydocker/clang-debian10:latest - a small image with preinstalled clang. + mydocker/clang-debian12:latest - a small image with preinstalled clang. Please note that this example produces a not very useful installation, since it doesn't override CMake defaults, which produces a Debug and non-boostrapped version of clang. To get a 2-stage clang build, you could use this command: -$ ./build_docker_image.sh -s debian10 -d mydocker/clang-debian10 -t "latest" \ +$ ./build_docker_image.sh -s debian12 -d mydocker/clang-debian12 -t "latest" \ -p clang -i stage2-install-clang -i stage2-install-clang-resource-headers \ -- \ -DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \ @@ -110,7 +110,7 @@ while [[ $# -gt 0 ]]; do CMAKE_ENABLED_PROJECTS="$CMAKE_ENABLED_PROJECTS;$PROJ" shift 2 ;; - -c|--checksums) + --checksums) shift CHECKSUMS_FILE="$1" shift @@ -160,7 +160,7 @@ if [ $SEEN_INSTALL_TARGET -eq 0 ]; then exit 1 fi -SOURCE_DIR=$(dirname $0) +SOURCE_DIR=$(dirname "$0") if [ ! -d "$SOURCE_DIR/$IMAGE_SOURCE" ]; then echo "No sources for '$IMAGE_SOURCE' were found in $SOURCE_DIR" exit 1 diff --git a/llvm/utils/docker/debian10/Dockerfile b/llvm/utils/docker/debian12/Dockerfile similarity index 79% rename from llvm/utils/docker/debian10/Dockerfile rename to llvm/utils/docker/debian12/Dockerfile index b898c935b972..84123c439ace 100644 --- a/llvm/utils/docker/debian10/Dockerfile +++ b/llvm/utils/docker/debian12/Dockerfile @@ -1,4 +1,4 @@ -#===- llvm/utils/docker/debian10/build/Dockerfile -------------------------===// +#===- llvm/utils/docker/debian12/build/Dockerfile -------------------------===// # # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. # See https://llvm.org/LICENSE.txt for license information. @@ -6,8 +6,8 @@ # #===----------------------------------------------------------------------===// # Stage 1. Check out LLVM source code and run the build. -FROM launcher.gcr.io/google/debian10:latest as builder -LABEL maintainer "LLVM Developers" +FROM launcher.gcr.io/google/debian12:latest AS builder +LABEL maintainer="LLVM Developers" # Install build dependencies of llvm. # First, Update the apt's source list and include the sources of the packages. RUN grep deb /etc/apt/sources.list | \ @@ -19,8 +19,8 @@ RUN apt-get update && \ rm -rf /var/lib/apt/lists/* # Install a newer ninja release. It seems the older version in the debian repos # randomly crashes when compiling llvm. -RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \ - echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \ +RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip" && \ + echo "6f98805688d19672bd699fbbfa2c2cf0fc054ac3df1f0e6a47664d963d530255 ninja-linux.zip" \ | sha256sum -c && \ unzip ninja-linux.zip -d /usr/local/bin && \ rm ninja-linux.zip @@ -37,11 +37,11 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg # Stage 2. Produce a minimal release image with build results. -FROM launcher.gcr.io/google/debian10:latest -LABEL maintainer "LLVM Developers" +FROM launcher.gcr.io/google/debian12:latest +LABEL maintainer="LLVM Developers" # Install packages for minimal useful image. RUN apt-get update && \ - apt-get install -y --no-install-recommends libstdc++-7-dev binutils && \ + apt-get install -y --no-install-recommends binutils && \ rm -rf /var/lib/apt/lists/* # Copy build results of stage 1 to /usr/local. COPY --from=builder /tmp/clang-install/ /usr/local/ diff --git a/llvm/utils/docker/example/Dockerfile b/llvm/utils/docker/example/Dockerfile index ebfb0e49c82c..197716fa7bdf 100644 --- a/llvm/utils/docker/example/Dockerfile +++ b/llvm/utils/docker/example/Dockerfile @@ -10,9 +10,9 @@ # Stage 1. Check out LLVM source code and run the build. # FIXME: Replace 'ubuntu' with your base image -FROM ubuntu as builder +FROM ubuntu AS builder # FIXME: Change maintainer name -LABEL maintainer "Maintainer " +LABEL maintainer="Maintainer " # FIXME: Install llvm/clang build dependencies here. Including compiler to # build stage1, cmake, subversion, ninja, etc. @@ -31,7 +31,7 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg # FIXME: Replace 'ubuntu' with your base image. FROM ubuntu # FIXME: Change maintainer name. -LABEL maintainer "Maintainer " +LABEL maintainer="Maintainer " # FIXME: Install all packages you want to have in your release container. # A minimal useful installation should include at least libstdc++ and binutils. diff --git a/llvm/utils/docker/nvidia-cuda/Dockerfile b/llvm/utils/docker/nvidia-cuda/Dockerfile index 91ad53c57b16..035c5825df0f 100644 --- a/llvm/utils/docker/nvidia-cuda/Dockerfile +++ b/llvm/utils/docker/nvidia-cuda/Dockerfile @@ -6,11 +6,11 @@ # #===----------------------------------------------------------------------===// # Stage 1. Check out LLVM source code and run the build. -FROM nvidia/cuda:8.0-devel as builder -LABEL maintainer "LLVM Developers" +FROM nvidia/cuda:12.6.3-devel-ubuntu24.04 AS builder +LABEL maintainer="LLVM Developers" # Install llvm build dependencies. RUN apt-get update && \ - apt-get install -y --no-install-recommends ca-certificates cmake python \ + apt-get install -y --no-install-recommends ca-certificates cmake 2to3 python-is-python3 \ subversion ninja-build git && \ rm -rf /var/lib/apt/lists/* @@ -26,8 +26,8 @@ RUN /tmp/scripts/build_install_llvm.sh --to /tmp/clang-install ${buildscript_arg # Stage 2. Produce a minimal release image with build results. -FROM nvidia/cuda:8.0-devel -LABEL maintainer "LLVM Developers" +FROM nvidia/cuda:12.6.3-devel-ubuntu24.04 +LABEL maintainer="LLVM Developers" # Copy clang installation into this container. COPY --from=builder /tmp/clang-install/ /usr/local/ # C++ standard library and binutils are already included in the base package. diff --git a/llvm/utils/docker/scripts/checkout.sh b/llvm/utils/docker/scripts/checkout.sh index 8fe7c6141d86..677fa61e6ae5 100755 --- a/llvm/utils/docker/scripts/checkout.sh +++ b/llvm/utils/docker/scripts/checkout.sh @@ -18,7 +18,7 @@ Checkout git sources into /tmp/clang-build/src. Used inside a docker container. Available options: -h|--help show this help message -b|--branch git branch to checkout, i.e. 'main', - 'release/10.x' + 'release/19.x' (default: 'main') -r|--revision git revision to checkout -c|--cherrypick revision to cherry-pick. Can be specified multiple times. @@ -83,7 +83,7 @@ function apply_cherrypicks() { # This function is always called on a sorted list of cherrypicks. for CHERRY_REV in $CHERRYPICKS; do echo "Cherry-picking $CHERRY_REV into $CHECKOUT_DIR" - EMAIL="someone@somewhere.net" git cherry-pick $CHERRY_REV + EMAIL="someone@somewhere.net" git cherry-pick "$CHERRY_REV" done popd @@ -97,7 +97,7 @@ mkdir -p "$CLANG_BUILD_DIR/src" CHECKOUT_DIR="$CLANG_BUILD_DIR/src" echo "Checking out https://github.com/llvm/llvm-project.git to $CHECKOUT_DIR" -git clone -b $LLVM_BRANCH --single-branch \ +git clone -b "$LLVM_BRANCH" --single-branch \ "https://github.com/llvm/llvm-project.git" \ "$CHECKOUT_DIR" @@ -114,7 +114,7 @@ CHECKSUMS_FILE="/tmp/checksums/checksums.txt" if [ -f "$CHECKSUMS_FILE" ]; then echo "Validating checksums for LLVM checkout..." - python "$(dirname $0)/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \ + python "$(dirname "$0")/llvm_checksum/llvm_checksum.py" -c "$CHECKSUMS_FILE" \ --partial --multi_dir "$CLANG_BUILD_DIR/src" else echo "Skipping checksumming checks..."