Merge pull request #928 from ktock/build-doc

Add a document about setting up `nerdctl build` with BuildKit

Author: AkihiroSuda

README.md

@@ -100,7 +100,7 @@ Binaries are available here: https://github.com/containerd/nerdctl/releases
 In addition to containerd, the following components should be installed:
 - [CNI plugins](https://github.com/containernetworking/plugins): for using `nerdctl run`.
    - v1.1.0 or later is highly recommended. Older versions require extra [CNI isolation plugin](https://github.com/AkihiroSuda/cni-isolation) for isolating bridge networks (`nerdctl network create`).
-- [BuildKit](https://github.com/moby/buildkit) (OPTIONAL): for using `nerdctl build`. BuildKit daemon (`buildkitd`) needs to be running.
+- [BuildKit](https://github.com/moby/buildkit) (OPTIONAL): for using `nerdctl build`. BuildKit daemon (`buildkitd`) needs to be running. See also [the document about setting up BuildKit](./docs/build.md).
 - [RootlessKit](https://github.com/rootless-containers/rootlesskit) and [slirp4netns](https://github.com/rootless-containers/slirp4netns) (OPTIONAL): for [Rootless mode](./docs/rootless.md)
    - RootlessKit needs to be v0.10.0 or later. v0.14.1 or later is recommended.
    - slirp4netns needs to be v0.4.0 or later. v1.1.7 or later is recommended.
@@ -749,7 +749,7 @@ Usage: `nerdctl unpause CONTAINER [CONTAINER...]`
 ### :whale: nerdctl build
 Build an image from a Dockerfile.
 
-:information_source: Needs buildkitd to be running.
+:information_source: Needs buildkitd to be running. See also [the document about setting up `nerdctl build` with BuildKit](./docs/build.md).
 
 Usage: `nerdctl build [OPTIONS] PATH`
 
@@ -1459,6 +1459,7 @@ Basic features:
 - [`./docs/compose.md`](./docs/compose.md):   Compose
 - [`./docs/rootless.md`](./docs/rootless.md): Rootless mode
 - [`./docs/cni.md`](./docs/cni.md): CNI for containers network
+- [`./docs/build.md`](./docs/build.md): `nerdctl build` with BuildKit
 
 Advanced features:
 - [`./docs/stargz.md`](./docs/stargz.md):     Lazy-pulling using Stargz Snapshotter

docs/build.md

@@ -0,0 +1,100 @@
+# Setting up `nerdctl build` with BuildKit
+
+`nerdctl build` (and `nerdctl compose build`) relies on [BuildKit](https://github.com/moby/buildkit).
+To use it, you need to set up BuildKit.
+
+BuildKit has 2 types of backends.
+
+- **containerd worker**: BuildKit relies on containerd to manage containers and images, etc. containerd needs to be up-and-running on the host.
+- **OCI worker**: BuildKit manages containers and images, etc. containerd isn't needed. This worker relies on runc for container execution.
+
+You need to set up BuildKit with either of the above workers.
+
+Note that OCI worker cannot access base images (`FROM` images in Dockerfiles) managed by containerd.
+Thus you cannot let `nerdctl build` use containerd-managed images as the base image.
+They include images previously built using `nerdctl build`.
+
+For example, the following build `bar` fails with OCI worker because it tries to use the previously built and containerd-managed image `foo`.
+
+```console
+$ mkdir -p /tmp/ctx && cat <<EOF > /tmp/ctx/Dockerfile
+FROM ghcr.io/stargz-containers/ubuntu:20.04-org
+RUN echo hello
+EOF
+$ nerdctl build -t foo /tmp/ctx
+$ cat <<EOF > /tmp/ctx/Dockerfile
+FROM foo
+RUN echo bar
+EOF
+$ nerdctl build -t bar /tmp/ctx
+```
+
+This limitation can be avoided using containerd worker as mentioned later.
+
+## Setting up BuildKit with containerd worker
+
+### Rootless (requires BuildKit v0.10.0 or later)
+
+```
+$ CONTAINERD_NAMESPACE=default containerd-rootless-setuptool.sh install-buildkit-containerd
+```
+
+`containerd-rootless-setuptool.sh` is aware of `CONTAINERD_NAMESPACE` and `CONTAINERD_SNAPSHOTTER` envvars.
+It installs buildkitd to the specified containerd namespace.
+This allows BuildKit using containerd-managed images in that namespace as the base image.
+Note that BuildKit can't use images in other namespaces as of now.
+
+If `CONTAINERD_NAMESPACE` envvar is not specified, this script configures buildkitd to use "buildkit" namespace (not "default" namespace).
+
+You can install an additional buildkitd process in a different namespace by executing this script with specifying the namespace with `CONTAINERD_NAMESPACE`.
+
+BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit-$CONTAINERD_NAMESPACE/buildkitd.sock` if `CONTAINERD_NAMESPACE` is specified.
+If `CONTAINERD_NAMESPACE` is not specified, that location will be `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`.
+
+### Rootful
+
+```
+$ sudo systemctl enable --now buildkit
+```
+
+Then add the following configuration to `/etc/buildkit/buildkitd.toml` to enable containerd worker.
+
+```toml
+[worker.oci]
+  enabled = false
+
+[worker.containerd]
+  enabled = true
+  # namespace should be "k8s.io" for Kubernetes (including Rancher Desktop)
+  namespace = "default"
+```
+
+## Setting up BuildKit with OCI worker
+
+### Rootless
+
+```
+$ containerd-rootless-setuptool.sh install-buildkit
+```
+
+As mentioned in the above, BuildKit with this configuration cannot use images managed by containerd.
+They include images previously built with `nerdctl build`.
+
+BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`.
+
+### rootful
+
+```
+$ sudo systemctl enable --now buildkit
+```
+
+## Which BuildKit socket will nerdctl use?
+
+You can specify BuildKit address for `nerdctl build` using `--buildkit-host` flag or `BUILDKIT_HOST` envvar.
+When BuildKit address isn't specified, nerdctl tries some default BuildKit addresses the following order and uses the first available one.
+
+- `<runtime directory>/buildkit-<current namespace>/buildkitd.sock`
+- `<runtime directory>/buildkit-default/buildkitd.sock`
+- `<runtime directory>/buildkit/buildkitd.sock`
+
+For example, if you run rootless nerdctl with `test` containerd namespace, it tries to use `$XDG_RUNTIME_DIR/buildkit-test/buildkitd.sock` by default then try to fall back to `$XDG_RUNTIME_DIR/buildkit-default/buildkitd.sock` and `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`