Skip to content

Commit 7749842

Browse files
kkaeferkkaefer
committed
rework Docker image build system
1 parent 08023be commit 7749842

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+457
-487
lines changed

‎.circleci/config.yml

+17-102
Original file line numberDiff line numberDiff line change
@@ -1,105 +1,20 @@
11
version: 2
22

3-
workflows:
4-
version: 2
5-
default:
6-
jobs:
7-
- start:
8-
type: approval
9-
- base:
10-
requires:
11-
- start
12-
- linux:
13-
requires:
14-
- base
15-
- linux-clang-3.8-libcxx:
16-
requires:
17-
- linux
18-
- linux-clang-3.9:
19-
requires:
20-
- linux
21-
- linux-clang-4:
22-
requires:
23-
- linux
24-
- linux-clang-5:
25-
requires:
26-
- linux
27-
- linux-gcc-4.9:
28-
requires:
29-
- linux
30-
- linux-gcc-5:
31-
requires:
32-
- linux
33-
- linux-gcc-5-qt-4:
34-
requires:
35-
- linux-gcc-5
36-
- linux-gcc-5-qt-5.9:
37-
requires:
38-
- linux-gcc-5
39-
- linux-gcc-6:
40-
requires:
41-
- linux
42-
- linux-gcc-7:
43-
requires:
44-
- linux
45-
- java:
46-
requires:
47-
- base
48-
- android-ndk-r18:
49-
requires:
50-
- java
51-
52-
defaults: &defaults
53-
working_directory: /app
54-
docker:
55-
- image: docker:17.10.0-ce-git
56-
steps:
57-
- checkout
58-
- setup_remote_docker:
59-
version: 17.10.0-ce # Some older versions don't support ARG before FROM in Dockerfiles
60-
- run:
61-
name: Log in to Docker Hub
62-
command: docker login -u mbgl -p ${DOCKER_PASS}
63-
- run:
64-
name: Revision ID
65-
command: echo "Building revision ${CIRCLE_SHA1:0:10}"
66-
- run:
67-
name: Build image
68-
command: docker build -t mbgl/${CIRCLE_SHA1:0:10}:${CIRCLE_STAGE} --build-arg REV=${CIRCLE_SHA1:0:10} ${CIRCLE_STAGE}
69-
no_output_timeout: 30m
70-
- run:
71-
name: Push image
72-
command: docker push mbgl/${CIRCLE_SHA1:0:10}:${CIRCLE_STAGE}
73-
74-
753
jobs:
76-
base:
77-
<<: *defaults
78-
79-
linux:
80-
<<: *defaults
81-
linux-clang-3.8-libcxx:
82-
<<: *defaults
83-
linux-clang-3.9:
84-
<<: *defaults
85-
linux-clang-4:
86-
<<: *defaults
87-
linux-clang-5:
88-
<<: *defaults
89-
linux-gcc-4.9:
90-
<<: *defaults
91-
linux-gcc-5:
92-
<<: *defaults
93-
linux-gcc-5-qt-4:
94-
<<: *defaults
95-
linux-gcc-5-qt-5.9:
96-
<<: *defaults
97-
linux-gcc-6:
98-
<<: *defaults
99-
linux-gcc-7:
100-
<<: *defaults
101-
102-
java:
103-
<<: *defaults
104-
android-ndk-r18:
105-
<<: *defaults
4+
build:
5+
working_directory: /app
6+
docker:
7+
- image: docker:18.05.0-ce-git
8+
steps:
9+
- checkout
10+
- run:
11+
name: Install dependencies
12+
command: apk add --no-progress bash
13+
- setup_remote_docker:
14+
version: 18.05.0-ce
15+
- run:
16+
name: Log in to Docker Hub
17+
command: docker login -u mbgl -p ${DOCKER_PASS}
18+
- run:
19+
name: Build Docker images
20+
command: scripts/build.sh

‎README.md

+6-28
Original file line numberDiff line numberDiff line change
@@ -1,45 +1,23 @@
11
This repository contains Docker files for running [Mapbox GL](https://github.com/mapbox/mapbox-gl-native) integration tests on [CircleCI](https://circleci.com/gh/mapbox/mapbox-gl-native).
22

3-
4-
53
### Guidelines
64

7-
* ⚠️ **Never overwrite published tags.** Instead, we use a revision system, using a Docker Hub repository for every set of images. The name of the repository is the 10 first characters of the Git SHA.
8-
* **Anchor images in hierarchy.** We use a hierarchy of images so that shared components (e.g. curl/zip/node) are only contained in one layer. CircleCI caches images, and using a shared base image means that we'll have to download fewer images from Docker Hub. See the hierarchy below.
5+
* ⚠️ **Never overwrite published tags.** Instead, we use a revision system, using a Docker Hub repository for every set of images. The name of the repository is the name of the image, and the versions is the first 10 bytes of the Git SHA.
96
* **Do not delete old images from Docker Hub**. Retaining old images on Docker Hub means that we can still build older branches.
107
* **Reduce ephemeral/temporary data** by combining multiple commands into one `RUN` command. Unfortunately, Circle CI doesn't support `--squash`, so we'll have to make sure that we're not creating unused files. E.g. when you run `apt-get update`, also run `rm -rf /var/lib/apt/lists/*` in the same step to remove the repository metadata.
118

12-
13-
14-
### Hierarchy
15-
16-
* **ubuntu:16.04** (~130MB)
17-
* **base**: curl/zip/node/python/pip (~800MB)
18-
* **linux**: build tools, OpenGL header + runtime (~190MB)
19-
* **linux-clang-3.9** (~390MB)
20-
* **linux-clang-4** (~470MB)
21-
* **linux-gcc-4.9** (~100MB)
22-
* **linux-gcc-5** (0 MB)
23-
* **linux-gcc-6** (~500MB)
24-
* ... (other compilers)
25-
* **java**: Java runtime, JDK, Google Cloud SDK (~330MB)
26-
* **android-ndk-r16**: Android NDK r16 (~2.35GB)
27-
* ... (other NDK images)
28-
29-
30-
319
### Adding a new image
3210

33-
When you want to add a new image, e.g. a new compiler version or a new Android NDK version, duplicate or rename an existing folder, and change the `Dockerfile` in it to suite your needs. There's no need to keep previous versions of e.g. the Android NDK around. Legacy branches of mapbox-gl-native will use legacy Docker images that contain the legacy NDKs.
11+
When you want to add a new image, e.g. a new compiler version or a new Android NDK version, duplicate an existing file in the `images` directory, annd change it to suit your needs. There's usually no need to keep previous versions of e.g. the Android NDK around. Legacy branches of mapbox-gl-native will use legacy Docker images that contain the legacy NDKs.
3412

3513
### Workflow
3614

37-
Whenever an image in the hierarchy changes, we're completely rebuilding the entire image hierarchy to benefit from Circle's Docker image caching. Builds are triggered when you start a **pull request**. However, builds are paused, and you'll have to manually click on the Resume button on Circle CI to start building. We do this to prevent building images too frequently, or when merging changes to master.
15+
To trigger a build, include `[docker:imagename]` in the commit message and push to Github. Only the most recent commit of a branch will be built.
3816

3917
Every set of images has a "Revision ID", which is based on the first 10 characters of the Git SHA. You can also see this ID by looking into the "Revision ID" step on any Circle CI build in the workflow.
4018

41-
When the workflow you're building **fails**, please remove the associated repository from Docker Hub to prevent heaps of unused Docker images piling up.
19+
When the workflow you're building **fails**, please remove images that you generated but don't need anymore from Docker Hub to prevent heaps of unused Docker images piling up.
4220

43-
Once a set of images completes building, it's time to update mapbox-gl-native. To do so, create a pull request that changes the Revision ID to the new one. If it succeeds, it's time "merge" the pull request in this repository that changed the image files. Instead of creating merge commits or merges via the GitHub UI, please do a fast-forward merge by switching to the master branch and running `git merge --ff-only <branchname>`. This allows use to remain a 1:1 relation between the Docker image Revision IDs and the Commit SHAs in this repository.
21+
### Local development
4422

45-
To reduce the number of image updates, and thus the number of images we have to keep on Docker Hub, we'll collect a few changes, merging everything into one branch before merging it to master.
23+
To speed up local development, you can run `scripts/apt-proxy.sh` to launch a local APT proxy that caches package downloads. Then run `./build.sh <imagename>` to build the image locally.

‎base/Dockerfile

-45
This file was deleted.

‎base/libsysconfcpus.so

-12.8 KB
Binary file not shown.

‎base/nodesource-v8.list

-1
This file was deleted.

‎base/nodesource.gpg

-2.15 KB
Binary file not shown.

‎build.sh

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
#!/usr/bin/env bash
2+
3+
set -eu
4+
cd "$(dirname "${BASH_SOURCE[0]}")"
5+
6+
PUSH=false
7+
ID="${CIRCLE_SHA1:-latest}"
8+
9+
function usage {
10+
echo "Usage: $0 [args] <image>"
11+
echo ""
12+
echo "Arguments:"
13+
echo " -p, --push Pushes the resulting image to Docker Hub (only for builds with an ID)"
14+
echo " -i, --id Specifies the ID of the image. Defaults to \`${CIRCLE_SHA1:-latest}\`"
15+
exit 1
16+
}
17+
18+
declare -a ARGS
19+
while [ $# -gt 0 ]; do
20+
case "$1" in
21+
-p|--push)
22+
PUSH=true
23+
shift
24+
;;
25+
-i|--id)
26+
if [ -z "${2:-}" ]; then
27+
echo -e "\033[31m$1 is missing an argument\033[0m\n"
28+
usage
29+
fi
30+
ID="${2}"
31+
shift
32+
shift
33+
;;
34+
-*)
35+
echo "Unknown argument $1"
36+
usage
37+
;;
38+
*)
39+
ARGS+=("$1")
40+
shift
41+
;;
42+
esac
43+
done
44+
if [ ${#ARGS[@]} -ne 0 ]; then
45+
set -- "${ARGS[@]}"
46+
fi
47+
48+
if [ -z "${1:-}" ]; then
49+
usage
50+
fi
51+
IMAGE="$1"
52+
53+
if [ ! -r "images/$IMAGE" ]; then
54+
echo "Can't find image $IMAGE"
55+
exit 1
56+
fi
57+
58+
if [ $PUSH = true ] && [ "$ID" == "latest" ]; then
59+
echo -e "\033[31mCan't push images without a named ID. Try setting CIRCLE_SHA1 to the commit sha\033[0m\n"
60+
usage
61+
fi
62+
63+
# Add HTTP proxy for local testing if there is an apt-proxy running
64+
# Run scripts/apt-proxy.sh locally to start.
65+
PREFIX=
66+
if nc -z localhost 3142 2>/dev/null; then
67+
PREFIX=$'\n''RUN echo "Acquire::http { Proxy \\"http://host.docker.internal:3142\\"; };" >> /etc/apt/apt.conf.d/01proxy'
68+
fi
69+
70+
# Add prefix to the Docker file and pipe it into stdin
71+
sed -e '/^FROM /a\'"$PREFIX" "images/$IMAGE" | docker build -t "mbgl/$IMAGE:${ID:0:10}" images -f -
72+
73+
if [ $PUSH = true ]; then
74+
docker tag "mbgl/$IMAGE:${ID:0:10}" "mbgl/$IMAGE:latest"
75+
docker push "mbgl/$IMAGE:${ID:0:10}"
76+
docker push "mbgl/$IMAGE:latest"
77+
fi

‎images/android-base

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
FROM ubuntu:xenial-20181005
2+
ENV DEBIAN_FRONTEND="noninteractive"
3+
4+
# Add base filesystems
5+
ADD filesystem/libsysconfcpus.tar.gz \
6+
filesystem/apt-google-cloud-sdk.tar.gz \
7+
/
8+
9+
# libsysconfcpus.so can be used to alter the number of CPUs reported by the system
10+
ENV LD_PRELOAD=/usr/lib/libsysconfcpus.so
11+
12+
# Add apt keys
13+
RUN set -eu \
14+
&& (find /tmp/*.gpg | xargs -n1 apt-key add) \
15+
&& rm /tmp/*.gpg
16+
17+
# Install base packages
18+
RUN set -eu \
19+
&& apt-get update \
20+
&& apt-get -y install git curl make python zip unzip ccache openjdk-8-jdk-headless \
21+
&& rm -rf /var/lib/apt/lists/*
22+
23+
# Install Google Cloud SDK
24+
RUN set -eu \
25+
&& apt-get update \
26+
&& apt-get -y install google-cloud-sdk \
27+
&& rm -rf /var/lib/apt/lists/*
28+
29+
# Install awscli
30+
RUN set -eu \
31+
&& (curl https://bootstrap.pypa.io/get-pip.py | python) \
32+
&& pip install awscli
33+
34+
# Install Node.js
35+
# Check https://nodejs.org/en/download/releases/ for releases
36+
ENV NVM_DIR="/root/.nvm" \
37+
PATH="/root/.nvm/versions/node/v10.13.0/bin:$PATH"
38+
RUN set -e \
39+
&& git -c advice.detachedHead=false clone https://github.com/creationix/nvm.git -b v0.33.11 --depth=1 "$NVM_DIR" \
40+
&& . "$NVM_DIR/nvm.sh" \
41+
&& nvm install v10.13.0
42+
43+
ENV ANDROID_NDK_HOME=/android/sdk/ndk-bundle \
44+
ANDROID_HOME=/android/sdk \
45+
GRADLE_OPTS=-Dorg.gradle.daemon=false

‎android-ndk-r18/Dockerfile ‎images/android-ndk-r17c

+7-15
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,36 @@
1-
ARG REV=latest
2-
FROM mbgl/${REV}:java
3-
4-
ENV ANDROID_NDK_HOME=/android/sdk/ndk-bundle \
5-
ANDROID_HOME=/android/sdk \
6-
GRADLE_OPTS=-Dorg.gradle.daemon=false
7-
8-
# --------------------------------------------------------------------------------------------------
9-
# Install Android SDK
1+
FROM mbgl/android-base
102

113
WORKDIR /android/sdk
124

5+
# Install Android SDK
136
# Use most recent version from https://developer.android.com/studio/index.html#command-tools
147
# and update the checksum.
158
RUN set -eu \
169
&& curl -L --retry 3 https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip -o tools.zip \
1710
&& (echo "92ffee5a1d98d856634e8b71132e8a95d96c83a63fde1099be3d86df3106def9 tools.zip" | sha256sum -c) \
1811
&& unzip -q tools.zip && rm tools.zip
1912

20-
# --------------------------------------------------------------------------------------------------
2113
# Install Android NDK
22-
2314
# Use desired version from https://developer.android.com/ndk/downloads/index.html
2415
# and update the checksum.
2516
RUN set -eu \
26-
&& curl -L --retry 3 https://dl.google.com/android/repository/android-ndk-r18-linux-x86_64.zip -o ndk.zip \
27-
&& (echo "2ac2e8e1ef73ed551cac3a1479bb28bd49369212 ndk.zip" | sha1sum -c) \
17+
&& curl -L --retry 3 https://dl.google.com/android/repository/android-ndk-r17c-linux-x86_64.zip -o ndk.zip \
18+
&& (echo "12cacc70c3fd2f40574015631c00f41fb8a39048 ndk.zip" | sha1sum -c) \
2819
&& unzip -q ndk.zip && rm ndk.zip && mv android-ndk-r* ndk-bundle
2920

30-
# --------------------------------------------------------------------------------------------------
3121
# Install dependencies and build tools
3222

3323
RUN set -eu \
3424
&& mkdir -p "${ANDROID_HOME}/licenses" \
3525
&& echo "d56f5187479451eabf01fb78af6dfcb131a6481e" > "${ANDROID_HOME}/licenses/android-sdk-license" \
3626
&& tools/bin/sdkmanager \
3727
"platform-tools" \
28+
"platforms;android-26" \
29+
"build-tools;26.0.3" \
3830
"platforms;android-27" \
3931
"build-tools;27.0.3" \
4032
"platforms;android-28" \
41-
"build-tools;28.0.3" \
33+
"build-tools;28.0.2" \
4234
"extras;android;m2repository" \
4335
"patcher;v4" \
4436
"extras;google;m2repository" \

0 commit comments

Comments
 (0)