Cross compile in the direct way

[Xuanwo]

Background

Databend used cross for cross-compilation for some time. But we have met many problems:

• Newly released bendctl started failed for libssl.so.1.0.0 not found #3423 (Because we build arm64 version on the ubuntu 16.04)
• Users report databend depends on GLIBC_2.29, but he only has GLIBC_2.28. (Because we build amd64 version on github hosted action runners which is ubuntu 20.04)
• cross itself is lack of maintenance, and will be transferred out of rust embedded team: RFC for moving cross to its own organisation rust-embedded/wg#590

Is it really that hard to cross-compile? You know, this is a problem that has existed since the beginning of the C language.

So I did some research. Finally, I found out that: This problem would be particularly simple if we only needed to support specific architectures. For example, let's say we only support Debian architectures that support cross-compilation.

Setup

Let me show you how I support cross build aarch64-unknown-linux-gnu on x86_64:

We can use docker to prevent installing too many dependencies on our host machine:

FROM rust:latest

RUN dpkg --add-architecture arm64 && \
    apt-get update && \
    apt-get install -y libc6-arm64-cross libc6-dev-arm64-cross gcc-aarch64-linux-gnu && \
    apt-get install -y libdbus-1-dev libdbus-1-dev:arm64  && \
    apt-get install -y libssl-dev libssl-dev:arm64 zlib1g-dev zlib1g-dev:arm64 && \
    apt-get clean

ENV PKG_CONFIG_ALLOW_CROSS=1
COPY cargo-config /usr/local/cargo/config

Let me explain them one by one.

FROM rust:latest

Use the rust upstream image, so that we don't need to set the basic rust environment.

RUN dpkg...

Install cross compile required deps, as explained in #3371

ENV PKG_CONFIG_ALLOW_CROSS=1

Make pkg-config allow cross-build (Yes, we have set up the dependencies)

COPY cargo-config /usr/local/cargo/config

We need to use the correct linker, the config content is:

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

Showtime

So we can build databend now:

podman run --network=host --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp test-rust bash -c "rustup target add aarch64-unknown-linux-gnu && rustup component add rustfmt && cargo build --target aarch64-unknown-linux-gnu"

• --network=host use the host network so my HTTP proxy will works correctly.
• --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp borrowed from rust upstream image.

How do you feel?

Next step

• Maybe we can remove the dependence on cross?
• Maybe we can build our own images to build all platform (on linux?) binaries?
• Maybe we can provide a simple wrapper like cross?