Skip to content

Commit b1469b4

Browse files
staticfloatstaticfloat
authored
[GCCBootstrap_jll] Build using crosstool-ng (#4753)
In the beginning, I wanted to compile a nice little standalone `GCC_jll` that could be hooked together with a `Glibc_jll` and a `Binutils_jll`, and a `LinuxKernelHeaders_jll`, etc... That work is sitting in [0] but bootstrapping is such a giant pain in the neck that I wanted to give up the complexity of bootstrapping to instead focus on simply building each product in isolation. This _vastly_ reduces the amount of work necessary to get things working, but it introduces a new dependency: we need a base C compiler and libc that are already compiled for the target platform. To be precise, we need a `build -> host` compiler in order to build a `host -> target` compiler. We already have one of those for all of our current platforms, so I could generate `GCC_jll` packages, but then when we want to add a new platform, we'd be in trouble. For this reason, I realized we'll never truly escape the bootstrapping problem. I thought about reverting back to the bootstrapping configuration we've had until now, but rebelled at the thought. Then I discovered crosstool-ng, and realized that I could separate concerns here: I can use ct-ng to build a working cross-toolchain for each target, then use that compiler to do a much simpler build for all of the other components. Therefore, I extract the work of getting a bootstrap build onto crosstool-ng, and then use that to do whatever other builds I want in the presence of a fully-functional cross-compiler. This also breaks the need for the initial bootstrap to be quite as restricted as the target toolchain. The GCC that we build technically doesn't need to run everywhere, it just needs to generate code that runs everywhere. We can build GCC against glibc 2.19, and then at build time have it link the code it generates against glibc 2.12.2, and that will work just fine for BB. The compiler may be using a newer glibc to run, but when it builds, it uses whatever glibc is placed within the target sysroot. This also means we don't need to do things like build GFortran as part of our bootstrap; we can build it later, in the simpler build script. In principle, we don't actually need a GCCBootstrap for all platforms. We only need a functional cross-compiler. Theoretically, we could use Clang to do the bootstrapping. But I'm not going to fully embrace that because I know that compiling Glibc with Clang is a pain, so for `*-linux-gnu` at the very least, we're not going to attempt that. On macOS and FreeBSD however, there is a possibility that we can use Clang as our "bootstrap compiler" in order to generate the actual GCC_jlls. [0] https://github.com/JuliaPackaging/Yggdrasil/tree/sf/gcc
1 parent ce2fb4a commit b1469b4

File tree

5 files changed

+426
-1
lines changed

5 files changed

+426
-1
lines changed

‎0_RootFS/Rootfs/bundled/libs/libc/download_libcs.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,5 @@ done
1212
for arch in i686 x86_64; do
1313
rm -rf musl-${arch}
1414
mkdir -p musl-${arch}
15-
curl -L "https://github.com/JuliaBinaryWrappers/Musl_jll.jl/releases/download/Musl-v1.2.2%2B1/Musl.v1.2.2.${arch}-linux-musl.tar.gz" | tar --wildcards --strip-components=1 -zxv -C musl-${arch} "lib*/"
15+
curl -L "https://github.com/JuliaBinaryWrappers/Musl_jll.jl/releases/download/Musl-v1.2.2%2B2/Musl.v1.2.2.${arch}-linux-musl.tar.gz" | tar --wildcards --strip-components=1 -zxv -C musl-${arch} "lib*/"
1616
done

‎G/GCCBootstrap/build_tarballs.jl

+130
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Note that this script can accept some limited command-line arguments, run
2+
# `julia build_tarballs.jl --help` to see a usage message.
3+
using BinaryBuilder, Pkg
4+
5+
name = "GCCBootstrap"
6+
version = v"9.4.0"
7+
8+
# Collection of sources required to complete build
9+
sources = [
10+
# crosstool-ng will provide the build script
11+
ArchiveSource("http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.25.0_rc1.tar.bz2",
12+
"8a839df71bea7b2c411447b41b917f2142482df171733ddb92d615cec2f0f43a"),
13+
14+
# We provide some configs for crostool-ng
15+
DirectorySource("./bundled"),
16+
17+
# crosstool-ng can download the files, but we'd rather download them ourselves
18+
FileSource("http://mirrors.kernel.org/gnu/gcc/gcc-9.4.0/gcc-9.4.0.tar.xz",
19+
"c95da32f440378d7751dd95533186f7fc05ceb4fb65eb5b85234e6299eb9838e"),
20+
FileSource("https://mirrors.kernel.org/gnu/mpfr/mpfr-4.1.0.tar.xz",
21+
"0c98a3f1732ff6ca4ea690552079da9c597872d30e96ec28414ee23c95558a7f"),
22+
FileSource("https://mirrors.kernel.org/gnu/mpc/mpc-1.2.1.tar.gz",
23+
"17503d2c395dfcf106b622dc142683c1199431d095367c6aacba6eec30340459"),
24+
FileSource("https://gcc.gnu.org/pub/gcc/infrastructure/isl-0.24.tar.bz2",
25+
"fcf78dd9656c10eb8cf9fbd5f59a0b6b01386205fe1934b3b287a0a1898145c0"),
26+
FileSource("https://mirrors.kernel.org/gnu/gmp/gmp-6.2.1.tar.xz",
27+
"fd4829912cddd12f84181c3451cc752be224643e87fac497b69edddadc49b4f2"),
28+
FileSource("http://mirrors.kernel.org/pub/linux/kernel/v4.x/linux-4.1.49.tar.xz",
29+
"ff2e0ea5c536650aef64447c3aaa49c1a25e8f1db4ec4f7da700d3176f512ba8"),
30+
FileSource("https://mirrors.kernel.org/gnu/glibc/glibc-2.19.tar.xz",
31+
"2d3997f588401ea095a0b27227b1d50cdfdd416236f6567b564549d3b46ea2a2"),
32+
FileSource("https://musl.libc.org/releases/musl-1.2.2.tar.gz",
33+
"9b969322012d796dc23dda27a35866034fa67d8fb67e0e2c45c913c3d43219dd"),
34+
FileSource("https://sourceforge.net/projects/mingw-w64/files/mingw-w64/mingw-w64-release/mingw-w64-v9.0.0.tar.bz2",
35+
"1929b94b402f5ff4d7d37a9fe88daa9cc55515a6134805c104d1794ae22a4181"),
36+
FileSource("https://github.com/madler/zlib/archive/refs/tags/v1.2.11.tar.gz",
37+
"629380c90a77b964d896ed37163f5c3a34f6e6d897311f1df2a7016355c45eff",
38+
filename="zlib-1.2.11.tar.gz"),
39+
FileSource("http://mirrors.kernel.org/gnu/ncurses/ncurses-6.2.tar.gz",
40+
"30306e0c76e0f9f1f0de987cf1c82a5c21e1ce6568b9227f7da5b71cbea86c9d"),
41+
FileSource("http://mirrors.kernel.org/gnu/libiconv/libiconv-1.16.tar.gz",
42+
"e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04"),
43+
FileSource("http://mirrors.kernel.org/gnu/gettext/gettext-0.21.tar.xz",
44+
"d20fcbb537e02dcf1383197ba05bd0734ef7bf5db06bdb241eb69b7d16b73192"),
45+
FileSource("http://mirrors.kernel.org/gnu/binutils/binutils-2.29.1.tar.xz",
46+
"e7010a46969f9d3e53b650a518663f98a5dde3c3ae21b7d71e5e6803bc36b577"),
47+
FileSource("http://mirrors.kernel.org/gnu/binutils/binutils-2.38.tar.xz",
48+
"e316477a914f567eccc34d5d29785b8b0f5a10208d36bbacedcc39048ecfe024"),
49+
]
50+
51+
# Bash recipe for building across all platforms
52+
script = raw"""
53+
cd ${WORKSPACE}/srcdir/crosstool-ng*/
54+
55+
# These tools will help us to bootstrap
56+
apk add build-base texinfo help2man ncurses-dev
57+
58+
# Copy in our extra patches for all packages
59+
for package in ${WORKSPACE}/srcdir/patches/*; do
60+
package="$(basename "${package}")"
61+
for version in ${WORKSPACE}/srcdir/patches/${package}/*; do
62+
version="$(basename "${version}")"
63+
if [ ! -d packages/${package}/${version} ]; then
64+
continue
65+
fi
66+
cp -v ${WORKSPACE}/srcdir/patches/${package}/${version}/* packages/${package}/${version}/
67+
done
68+
done
69+
70+
# Disable some checks that ct-ng performs
71+
export CT_ALLOW_BUILD_AS_ROOT_SURE=1
72+
73+
# Unset some things that BB automatically inserts into the environment,
74+
# but which crosstool-ng rightfully complains about.
75+
unset LD_LIBRARY_PATH
76+
for TOOL in CC CXX LD AS AR FC OBJCOPY OBJDUMP RANLIB STRIP LIPO MESON NM READELF; do
77+
unset "${TOOL}" "BUILD_${TOOL}" "${TOOL}_BUILD" "${TOOL}_FOR_BUILD" "HOST${TOOL}"
78+
done
79+
PATH="$(printf "%s" "$(echo -n "${PATH}" | tr ':' '\n' | grep -v "/opt")" | tr '\n' ':')"
80+
81+
82+
# Build crosstool-ng for the current host
83+
./configure --enable-local
84+
make -j${nproc}
85+
86+
# Generate the appropriate crosstool-ng config file for our current target
87+
${WORKSPACE}/srcdir/gen_config.sh > .config
88+
cat .config
89+
90+
# This takes our stripped-down config and fills out all the other options
91+
./ct-ng upgradeconfig
92+
93+
# Do the actual build!
94+
./ct-ng build
95+
96+
# Fix case-insensitivity problems in netfilter headers
97+
if [[ "${target}" == *linux* ]]; then
98+
NF="${prefix}/${target}/sysroot/usr/include/linux/netfilter"
99+
for NAME in CONNMARK DSCP MARK RATEEST TCPMSS; do
100+
mv "${NF}/xt_${NAME}.h" "${NF}/xt_${NAME}_.h"
101+
done
102+
for NAME in ECN TTL; do
103+
mv "${NF}_ipv4/ipt_${NAME}.h" "${NF}_ipv4/ipt_${NAME}_.h"
104+
done
105+
mv "${NF}_ipv6/ip6t_HL.h" "${NF}_ipv6/ip6t_HL_.h"
106+
fi
107+
108+
# Move licenses to the right spot
109+
mkdir -p /tmp/GCCBootstrap
110+
mv ${prefix}/share/licenses/* /tmp/GCCBootstrap
111+
mv /tmp/GCCBootstrap ${prefix}/share/licenses/
112+
113+
[[ -f "${bindir}/${target}-gcc" ]]
114+
"""
115+
116+
# These are the platforms we will build for by default, unless further
117+
# platforms are passed in on the command line
118+
platforms = filter(p -> Sys.islinux(p) || Sys.iswindows(p), supported_platforms())
119+
120+
# The products that we will ensure are always built
121+
products = Product[
122+
#ExecutableProduct("gcc", :gcc)
123+
]
124+
125+
# Dependencies that must be installed before this package can be built
126+
dependencies = Dependency[
127+
]
128+
129+
# Build the tarballs, and possibly a `build.jl` as well.
130+
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; julia_compat="1.6")

‎G/GCCBootstrap/bundled/gen_config.sh

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
#!/bin/bash
2+
3+
set -euo pipefail
4+
5+
# Start wtih things that are always true
6+
cat <<-EOF
7+
CT_CONFIG_VERSION=4
8+
9+
# Always build GCC v9.X
10+
CT_GCC_V_9=y
11+
12+
# Disable progress bar, it fills up our logs:
13+
CT_LOG_PROGRESS_BAR=n
14+
15+
# We use some experimental features, just enable them for all
16+
CT_EXPERIMENTAL=y
17+
18+
# Explicitly claim the '--build' triplet, so there's no confusion
19+
CT_BUILD="${MACHTYPE}"
20+
21+
# Tell ct-ng to not remove our prefix... this really confused me, as the
22+
# build would finish, but the resultant artifact was empty, because it
23+
# would have cleared out the symlink to the target-specific prefix. Orz.
24+
CT_PREFIX_DIR="${prefix}"
25+
CT_PREFIX_DIR_RO=n
26+
CT_RM_RF_PREFIX_DIR=n
27+
28+
# Don't download any tarballs; we should have taken care of that already
29+
CT_FORBID_DOWNLOAD=y
30+
CT_LOCAL_TARBALLS_DIR="${WORKSPACE}/srcdir"
31+
32+
# We always want to build g++
33+
CT_CC_LANG_CXX=y
34+
EOF
35+
36+
# Handle OS stuff
37+
case "${target}" in
38+
*linux*)
39+
cat <<-EOF
40+
# We're building against linux, and always use an older kernel
41+
# version so that we are maximally compatible.
42+
CT_KERNEL_LINUX=y
43+
CT_LINUX_V_4_1=y
44+
45+
# We don't like the 'unknown' for Linux
46+
CT_OMIT_TARGET_VENDOR=y
47+
CT_TARGET_VENDOR=
48+
EOF
49+
;;
50+
*mingw*)
51+
cat <<-EOF
52+
CT_TARGET_VENDOR="w64"
53+
CT_KERNEL_WINDOWS=y
54+
EOF
55+
;;
56+
*)
57+
echo "Unhandled OS '${target}'" >&2
58+
exit 1
59+
;;
60+
esac
61+
62+
# Handle arch stuff
63+
case "${target}" in
64+
arm*)
65+
echo "CT_ARCH_ARM=y"
66+
echo "CT_ARCH_FLOAT_HW=y"
67+
if [[ "${bb_full_target}" == armv6l* ]]; then
68+
echo "CT_ARCH_ARCH=\"armv6+fp\""
69+
elif [[ "${bb_full_target}" == armv7l* ]]; then
70+
echo "CT_ARCH_ARCH=\"armv7-a+fp\""
71+
else
72+
echo "ERROR: Unknown arm microarchitecture in ${bb_full_target}!" >&2
73+
exit 1
74+
fi
75+
;;
76+
aarch64*)
77+
echo "CT_ARCH_ARM=y"
78+
echo "CT_ARCH_64=y"
79+
;;
80+
i686*)
81+
echo "CT_ARCH_X86=y"
82+
echo "CT_ARCH_ARCH=\"pentium4\""
83+
;;
84+
x86_64*)
85+
echo "CT_ARCH_X86=y"
86+
echo "CT_ARCH_64=y"
87+
echo "CT_ARCH_ARCH=\"x86-64\""
88+
;;
89+
powerpc64le*)
90+
echo "CT_ARCH_POWERPC=y"
91+
echo "CT_ARCH_LE=y"
92+
echo "CT_ARCH_64=y"
93+
;;
94+
*)
95+
echo "ERROR: Unhandled arch '${target}'" >&2
96+
exit 1
97+
esac
98+
99+
# Handle libc stuff
100+
case "${target}" in
101+
*gnu*)
102+
echo "CT_GLIBC_V_2_19=y"
103+
;;
104+
*musl*)
105+
echo "CT_LIBC_MUSL=y"
106+
echo "CT_MUSL_v_1_2_2=y"
107+
;;
108+
*mingw*)
109+
echo "CT_LIBC_MINGW_W64=y"
110+
echo "CT_THREADS_POSIX=y"
111+
echo "CT_MINGW_W64_V_V9_0=y"
112+
;;
113+
*)
114+
echo "ERROR: Unhandled libc '${target}'" >&2
115+
exit 1
116+
;;
117+
esac

‎G/GCCBootstrap/bundled/patches/glibc/2.19/0100-sunrpc_on_musl.patch

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
diff --git a/sunrpc/rpc/types.h b/sunrpc/rpc/types.h
2+
index cf50141400..4adc625243 100644
3+
--- a/sunrpc/rpc/types.h
4+
+++ b/sunrpc/rpc/types.h
5+
@@ -68,7 +68,7 @@ typedef unsigned long rpcport_t;
6+
#include <sys/types.h>
7+
#endif
8+
9+
-#ifndef __u_char_defined
10+
+#if 0
11+
typedef __u_char u_char;
12+
typedef __u_short u_short;
13+
typedef __u_int u_int;
14+
@@ -78,7 +78,7 @@ typedef __u_quad_t u_quad_t;
15+
typedef __fsid_t fsid_t;
16+
# define __u_char_defined
17+
#endif
18+
-#ifndef __daddr_t_defined
19+
+#if 0
20+
typedef __daddr_t daddr_t;
21+
# if ! defined(caddr_t) && ! defined(__caddr_t_defined)
22+
typedef __caddr_t caddr_t;

0 commit comments

Comments
 (0)