theStack
diff --git a/Diff for: ‎contrib/gitian-descriptors/assign_DISTNAME
+1-1 b/Diff for: ‎contrib/gitian-descriptors/assign_DISTNAME
+1-1
diff --git a/Diff for: ‎contrib/guix/README.md
+104-5 b/Diff for: ‎contrib/guix/README.md
+104-5
diff --git a/Diff for: ‎contrib/guix/guix-build.sh renamed to ‎contrib/guix/guix-build
+76-33 b/Diff for: ‎contrib/guix/guix-build.sh renamed to ‎contrib/guix/guix-build
+76-33
@@ -4,7 +4,7 @@
 #
 # A helper script to be sourced into the gitian descriptors
 
-if RECENT_TAG="$(git describe --exact-match HEAD)"; then
+if RECENT_TAG="$(git describe --exact-match HEAD 2> /dev/null)"; then
     VERSION="${RECENT_TAG#v}"
 else
     VERSION="$(git rev-parse --short=12 HEAD)"
 
@@ -80,6 +80,50 @@ at the end of the `guix pull`)
 export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
 ```
 
+### Controlling the number of threads used by `guix` build commands
+
+By default, the scripts under `./contrib/guix` will invoke all `guix` build
+commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
+specified. However, astute manual readers will also notice that there is a
+`--max-jobs=` flag (which defaults to 1 if unspecified).
+
+Here is the difference between `--cores=` and `--max-jobs=`:
+
+> Note: When I say "derivation," think "package"
+
+`--cores=`
+
+  - controls the number of CPU cores to build each derivation. This is the value
+    passed to `make`'s `--jobs=` flag.
+
+`--max-jobs=`
+
+  - controls how many derivations can be built in parallel
+  - defaults to 1
+
+Therefore, the default is for `guix` build commands to build one derivation at a
+time, utilizing `$JOBS` threads.
+
+Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
+can also modify the value for `--max-jobs=` by specifying
+`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
+may want to set:
+
+```sh
+export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
+```
+
+Which allows for a maximum of 8 derivations to be built at the same time, each
+utilizing `$JOBS` threads.
+
+Or, if you'd like to avoid spurious build failures caused by issues with
+parallelism within a single package, but would still like to build multiple
+packages when the dependency graph allows for it, you may want to try:
+
+```sh
+export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
+```
+
 ## Usage
 
 ### As a Tool for Deterministic Builds
@@ -125,12 +169,16 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
   the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
   $HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
 
-* _**MAX_JOBS**_
+* _**JOBS**_
+
+  Override the number of jobs to run simultaneously, you might want to do so on
+  a memory-limited machine. This may be passed to:
+
+  - `guix` build commands as in `guix environment --cores="$JOBS"`
+  - `make` as in `make --jobs="$JOBS"`
+  - `xargs` as in `xargs -P"$JOBS"`
 
-  Override the maximum number of jobs to run simultaneously, you might want to
-  do so on a memory-limited machine. This may be passed to `make` as in `make
-  --jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
-  value of `nproc` outside the container)_
+  _(defaults to the value of `nproc` outside the container)_
 
 * _**SOURCE_DATE_EPOCH**_
 
@@ -217,6 +265,57 @@ To use dongcarl's substitute server for Bitcoin Core builds after having
 export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
 ```
 
+## Troubleshooting
+
+### Derivation failed to build
+
+When you see a build failure like below:
+
+```
+building /gnu/store/...-foo-3.6.12.drv...
+/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
+builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
+build of /gnu/store/...-foo-3.6.12.drv failed
+View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
+cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
+cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
+cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
+guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
+```
+
+It means that `guix` failed to build a package named `foo`, which was a
+dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
+line is not necessarily the root cause, the first "failed" line is.
+
+Most of the time, the build failure is due to a spurious test failure or the
+package's build system/test suite breaking when running multi-threaded. To
+rebuild _just_ this derivation in a single-threaded fashion:
+
+```sh
+$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
+```
+
+If the single-threaded rebuild stil did not succeed, you may need to dig deeper.
+You may view `foo`'s build logs in `less` like so (please replace paths with the
+path you see in the build failure output):
+
+```sh
+$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
+```
+
+`foo`'s build directory is also preserved and available at
+`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
+times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
+failure output for the most accurate, up-to-date information.
+
+#### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
+
+This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
+which rejects characters not present in the UTF-8 character code set. An example
+is ZFS with the utf8only=on option set.
+
+More information: https://bugs.python.org/issue37584
+
 ## FAQ
 
 ### How can I trust the binary installation?
 
@@ -2,22 +2,26 @@
 export LC_ALL=C
 set -e -o pipefail
 
+# Source the common prelude, which:
+#   1. Checks if we're at the top directory of the Bitcoin Core repository
+#   2. Defines a few common functions and variables
+#
+# shellcheck source=libexec/prelude.bash
+source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
+
+
 ###################
-## Sanity Checks ##
+## SANITY CHECKS ##
 ###################
 
 ################
-# Check 1: Make sure that we can invoke required tools
+# Required non-builtin commands should be invokable
 ################
-for cmd in git make guix cat mkdir curl; do
-    if ! command -v "$cmd" > /dev/null 2>&1; then
-        echo "ERR: This script requires that '$cmd' is installed and available in your \$PATH"
-        exit 1
-    fi
-done
+
+check_tools cat mkdir make git guix
 
 ################
-# Check 2: Make sure GUIX_BUILD_OPTIONS is empty
+# GUIX_BUILD_OPTIONS should be empty
 ################
 #
 # GUIX_BUILD_OPTIONS is an environment variable recognized by guix commands that
@@ -45,8 +49,9 @@ exit 1
 fi
 
 ################
-# Check 3: Make sure that we're not in a dirty worktree
+# The git worktree should not be dirty
 ################
+
 if ! git diff-index --quiet HEAD -- && [ -z "$FORCE_DIRTY_WORKTREE" ]; then
 cat << EOF
 ERR: The current git worktree is dirty, which may lead to broken builds.
@@ -60,27 +65,25 @@ Hint: To make your git worktree clean, You may want to:
          using a dirty worktree
 EOF
 exit 1
-else
-    GIT_COMMIT=$(git rev-parse --short=12 HEAD)
 fi
 
+mkdir -p "$VERSION_BASE"
+
 ################
-# Check 4: Make sure that build directories do not exist
+# Build directories should not exist
 ################
 
 # Default to building for all supported HOSTs (overridable by environment)
 export HOSTS="${HOSTS:-x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
                        x86_64-w64-mingw32
                        x86_64-apple-darwin18}"
 
-DISTSRC_BASE="${DISTSRC_BASE:-${PWD}}"
-
 # Usage: distsrc_for_host HOST
 #
 #   HOST: The current platform triple we're building for
 #
 distsrc_for_host() {
-    echo "${DISTSRC_BASE}/distsrc-${GIT_COMMIT}-${1}"
+    echo "${DISTSRC_BASE}/distsrc-${VERSION}-${1}"
 }
 
 # Accumulate a list of build directories that already exist...
@@ -106,12 +109,11 @@ for host in $hosts_distsrc_exists; do
 done
 exit 1
 else
-
     mkdir -p "$DISTSRC_BASE"
 fi
 
 ################
-# Check 5: When building for darwin, make sure that the macOS SDK exists
+# When building for darwin, the macOS SDK should exists
 ################
 
 for host in $HOSTS; do
@@ -128,13 +130,40 @@ for host in $HOSTS; do
     esac
 done
 
+################
+# Check that we can connect to the guix-daemon
+################
+
+cat << EOF
+Checking that we can connect to the guix-daemon...
+
+Hint: If this hangs, you may want to try turning your guix-daemon off and on
+      again.
+
+EOF
+if ! guix gc --list-failures > /dev/null; then
+cat << EOF
+
+ERR: Failed to connect to the guix-daemon, please ensure that one is running and
+     reachable.
+EOF
+exit 1
+fi
+
+# Developer note: we could use `guix repl` for this check and run:
+#
+#     (import (guix store)) (close-connection (open-connection))
+#
+# However, the internal API is likely to change more than the CLI invocation
+
+
 #########
-# Setup #
+# SETUP #
 #########
 
 # Determine the maximum number of jobs to run simultaneously (overridable by
 # environment)
-MAX_JOBS="${MAX_JOBS:-$(nproc)}"
+JOBS="${JOBS:-$(nproc)}"
 
 # Usage: host_to_commonname HOST
 #
@@ -152,7 +181,7 @@ host_to_commonname() {
 # Download the depends sources now as we won't have internet access in the build
 # container
 for host in $HOSTS; do
-  make -C "${PWD}/depends" -j"$MAX_JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
+    make -C "${PWD}/depends" -j"$JOBS" download-"$(host_to_commonname "$host")" ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
 done
 
 # Determine the reference time used for determinism (overridable by environment)
@@ -164,19 +193,29 @@ time-machine() {
     # shellcheck disable=SC2086
     guix time-machine --url=https://github.com/dongcarl/guix.git \
                       --commit=490e39ff303f4f6873a04bfb8253755bdae1b29c \
-                      --max-jobs="$MAX_JOBS" \
+                      --cores="$JOBS" \
                       --keep-failed \
+                      --fallback \
                       ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
                       ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_TIMEMACHINE_FLAGS} \
                       -- "$@"
 }
 
 # Make sure an output directory exists for our builds
-OUTDIR="${OUTDIR:-${PWD}/output}"
-[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
+OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"
+mkdir -p "$OUTDIR_BASE"
+
+# Usage: outdir_for_host HOST
+#
+#   HOST: The current platform triple we're building for
+#
+outdir_for_host() {
+    echo "${OUTDIR_BASE}/${1}"
+}
+
 
 #########
-# Build #
+# BUILD #
 #########
 
 # Function to be called when building for host ${1} and the user interrupts the
@@ -216,15 +255,15 @@ for host in $HOSTS; do
 
         # shellcheck disable=SC2030
 cat << EOF
-INFO: Building commit ${GIT_COMMIT:?not set} for platform triple ${HOST:?not set}:
+INFO: Building ${VERSION:?not set} for platform triple ${HOST:?not set}:
       ...using reference timestamp: ${SOURCE_DATE_EPOCH:?not set}
-      ...running at most ${MAX_JOBS:?not set} jobs
+      ...running at most ${JOBS:?not set} jobs
       ...from worktree directory: '${PWD}'
           ...bind-mounted in container to: '/bitcoin'
       ...in build directory: '$(distsrc_for_host "$HOST")'
           ...bind-mounted in container to: '$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")'
-      ...outputting in: '${OUTDIR:?not set}'
-          ...bind-mounted in container to: '/outdir'
+      ...outdirting in: '$(outdir_for_host "$HOST")'
+          ...bind-mounted in container to: '$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")'
 EOF
 
         # Run the build script 'contrib/guix/libexec/build.sh' in the build
@@ -299,24 +338,28 @@ EOF
                                  --no-cwd \
                                  --share="$PWD"=/bitcoin \
                                  --share="$DISTSRC_BASE"=/distsrc-base \
-                                 --share="$OUTDIR"=/outdir \
+                                 --share="$OUTDIR_BASE"=/outdir-base \
                                  --expose="$(git rev-parse --git-common-dir)" \
                                  ${SOURCES_PATH:+--share="$SOURCES_PATH"} \
                                  ${BASE_CACHE:+--share="$BASE_CACHE"} \
                                  ${SDK_PATH:+--share="$SDK_PATH"} \
-                                 --max-jobs="$MAX_JOBS" \
+                                 --cores="$JOBS" \
                                  --keep-failed \
+                                 --fallback \
+                                 --link-profile \
                                  ${SUBSTITUTE_URLS:+--substitute-urls="$SUBSTITUTE_URLS"} \
                                  ${ADDITIONAL_GUIX_COMMON_FLAGS} ${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
                                  -- env HOST="$host" \
-                                        MAX_JOBS="$MAX_JOBS" \
+                                        DISTNAME="$DISTNAME" \
+                                        JOBS="$JOBS" \
                                         SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
                                         ${V:+V=1} \
                                         ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
                                         ${BASE_CACHE:+BASE_CACHE="$BASE_CACHE"} \
                                         ${SDK_PATH:+SDK_PATH="$SDK_PATH"} \
                                         DISTSRC="$(DISTSRC_BASE=/distsrc-base && distsrc_for_host "$HOST")" \
-                                        OUTDIR=/outdir \
+                                        OUTDIR="$(OUTDIR_BASE=/outdir-base && outdir_for_host "$HOST")" \
+                                        DIST_ARCHIVE_BASE=/outdir-base/dist-archive \
                                       bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
     )
Original file line number	Diff line number	Diff line change
`@@ -4,7 +4,7 @@`
`4`	`4`	`#`
`5`	`5`	`# A helper script to be sourced into the gitian descriptors`
`6`	`6`
`7`		`-if RECENT_TAG="$(git describe --exact-match HEAD)"; then`
	`7`	`+if RECENT_TAG="$(git describe --exact-match HEAD 2> /dev/null)"; then`
`8`	`8`	`VERSION="${RECENT_TAG#v}"`
`9`	`9`	`else`
`10`	`10`	`VERSION="$(git rev-parse --short=12 HEAD)"`