Building with depends system from local git subtree directory fails

[ryanofsky]

Originally posted by @Sjors in bitcoin/bitcoin#31741 (review)

Building MULTIPROCESS=1 with depends now fails:

...
Postprocessing native_capnp...
Caching native_capnp...
Fetching from
curl: (3) URL rejected: No host part in the URL
Fetching from https://bitcoincore.org/depends-sources
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403
make: *** [/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22

Before commit 01452f8c0da58db549bc46d0cfa7de715344efc4 depends does build (as you would expect).

[ryanofsky]

Using bitcoin/bitcoin#31741 (https://github.com/ryanofsky/bitcoin/commits/pr/subtree.3) building works for me it would help to get more debug output.

Can you try to print some variables to see if they have expected values?

make MULTIPROCESS=1 print-native_capnp_download_file
make MULTIPROCESS=1 print-native_capnp_download_dir
make MULTIPROCESS=1 print-native_capnp_source
make MULTIPROCESS=1 print-native_libmultiprocess_local_dir
make MULTIPROCESS=1 print-native_libmultiprocess_fetched
make MULTIPROCESS=1 print-native_libmultiprocess_source

For me these look like:

native_capnp_download_file=capnproto-c++-1.1.0.tar.gz
native_capnp_download_dir=/home/russ/work/bitcoin/depends/work/download/native_capnp-1.1.0
native_capnp_source=/home/russ/work/bitcoin/depends/sources/capnproto-cxx-1.1.0.tar.gz
native_libmultiprocess_local_dir=../src/ipc/libmultiprocess
native_libmultiprocess_fetched=/home/russ/work/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
native_libmultiprocess_source=/home/russ/work/bitcoin/depends/sources/src-ipc-libmultiprocess.tar

It might also help to see the command line and complete output including V=1

[Sjors]

macOS 15.2 with Xcode 16.2

% make MULTIPROCESS=1 print-native_capnp_download_file
/bin/sh: command -v llvm-ranlib: No such file or directory
/bin/sh: command -v llvm-strip: No such file or directory
/bin/sh: command -v llvm-nm: No such file or directory
/bin/sh: command -v llvm-objdump: No such file or directory
/bin/sh: command -v dsymutil: No such file or directory
native_capnp_download_file=capnproto-c++-1.1.0.tar.gz

These warnings happen on master too, see bitcoin/bitcoin#30978

I'll omit them:

native_capnp_download_file=capnproto-c++-1.1.0.tar.gz
native_capnp_download_dir=/Users/sjors/dev/bitcoin/depends/work/download/native_capnp-1.1.0
native_capnp_source=/Users/sjors/dev/bitcoin/depends/sources/capnproto-cxx-1.1.0.tar.gz
native_libmultiprocess_local_dir=../src/ipc/libmultiprocess
native_libmultiprocess_fetched=/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
native_libmultiprocess_source=/Users/sjors/dev/bitcoin/depends/sources/src-ipc-libmultiprocess.tar

make MULTIPROCESS=1 NO_QT=1 NO_ZMQ=1 NO_WALLET=1 NO_USDT=1 V=1
...
mkdir -p /Users/sjors/dev/bitcoin/depends/sources/download-stamps /Users/sjors/dev/bitcoin/depends/sources
rm -f /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
touch /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
cd /Users/sjors/dev/bitcoin/depends/sources/download-stamps; ( test -f /Users/sjors/dev/bitcoin/depends/sources/ || (     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from  && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- ) ||     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from https://bitcoincore.org/depends-sources && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "https://bitcoincore.org/depends-sources/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- )))
Fetching from
curl: (3) URL rejected: No host part in the URL
Fetching from https://bitcoincore.org/depends-sources
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403
make: *** [/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22

Full log which includes the successfully built stuff: https://gist.github.com/Sjors/2b966733f511d984e298418e792835bc

[ryanofsky]

Thanks, I misread the original message and thought it was failing trying to download native_capnp sources, but actually looking later in the log it is failing trying to download the native_libmultiprocess sources. This is still wrong, because it should be using the subtree and not downloading anything. But at least it makes more sense.

For debugging, first thing it would be helpful to know if the two printed files exist:

native_libmultiprocess_source=/Users/sjors/dev/bitcoin/depends/sources/src-ipc-libmultiprocess.tar
native_libmultiprocess_fetched=/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-

before & after running the command. You should also be able to get in a clean state by deleting them before running it. The first file is supposed to contain a tarball of the git subtree, and second file is supposed to contain sha256 checksum file containing a hash of the tarball.

In your gist the first indication that there is a problem is line 537 where it appears make is trying to run the native_libmultiprocess_fetched build rule which is supposed to be skipped because the $(1)_fetched file .stamp_fetched-native_libmultiprocess-.hash should have already been created earlier on as part of computing the build ids.

It would also be helpful to see output of the following print command to see if build id is including a tarball hash:

make MULTIPROCESS=1 print-native_libmultiprocess_all_file_checksums

And a make --trace command which should reveal why make is trying to build a stamp_fetched-native_libmultiprocess-.hash file that should already exist.

make MULTIPROCESS=1 --trace native_libmultiprocess_fetched V=1

It would also be helpful if you can add some debug prints in code supposed to be creating the tarball and hash file:

--- a/depends/funcs.mk
+++ b/depends/funcs.mk
@@ -41,12 +41,15 @@ endef
 
 define int_get_local_dir
 $(eval $(1)_sha256_hash:=$(shell
+    set -x; \
+    echo >&2 HELLO HELLO CREATING $($(1)_fetched); \
     if ! [ -f $($(1)_fetched) ] || [ $($(1)_fetched) -ot $($(1)_local_dir) ]; then \
         mkdir -p $(dir $($(1)_fetched)); \
         $(build_TAR) -c --sort=name --mtime=0 --owner=0 --group=0 --numeric-owner -f $($(1)_source) -C $($(1)_local_dir) . && \
         $(build_SHA256SUM) $($(1)_source) > $($(1)_fetched); \
     fi; \
-    cut -d" " -f1 $($(1)_fetched); ))
+    cut -d" " -f1 $($(1)_fetched); \
+    cat >&2 $($(1)_fetched); ))
 $(eval $(1)_all_file_checksums+=$($(1)_sha256_hash))
 endef

This output should show up early on when running any of the commands above.

[Sjors]

For debugging, first thing it would be helpful to know if the two printed files exist:

Not before and not after.

With the patch applied, there's no HELLO in the output (with V=1).

native_libmultiprocess_all_file_checksums=8a0685199f5ff048f31c04cedb8bf13bc01d1177b96e08dd862846dcad812f4a ec2577614252326f889df1de97b9a457c03a9a94811048563c211a44496d8ba3 465c5fe67c7d36b72e808812716445d4d38d4b94734ca8d36a8639d12323878b 03ebe6e49b63a03435727b4c55d29f5e36ee5a1836e96456a2392b210e543b73 5ef5512589a90c1670dc231f6e996ef63af81827d03308b53a44c16591ec7c4a 7efbfcf0f23a34e643e8966fbcf78137f7a72e17c08e9b3d60730a023c0b16aa 0813e891aee723243545e1f5334622dafc861fd8abc2baace9e893e13330eca0 7ff2d66419eced83a5db9e06bd80f1139f53a8f9be95af1924abad6ba1ad045a 7261275dea35a4448176996fca08adc766445f7048ea2cacc15334d30f0e0ebe 

There's no --trace argument for make, did you mean --debug?

             Successfully remade target file `native_capnp'.
            File `/Users/sjors/dev/bitcoin/depends/built/aarch64-apple-darwin24.2.0/native_libmultiprocess/native_libmultiprocess--6dd32f3a9cb.tar.gz' does not exist.
             File `/Users/sjors/dev/bitcoin/depends/work/staging/aarch64-apple-darwin24.2.0/native_libmultiprocess/-6dd32f3a9cb/Users/sjors/dev/bitcoin/depends/aarch64-apple-darwin24.2.0/native/.stamp_postprocessed' does not exist.
              File `/Users/sjors/dev/bitcoin/depends/work/staging/aarch64-apple-darwin24.2.0/native_libmultiprocess/-6dd32f3a9cb/.stamp_staged' does not exist.
               File `/Users/sjors/dev/bitcoin/depends/work/build/aarch64-apple-darwin24.2.0/native_libmultiprocess/-6dd32f3a9cb/./.stamp_built' does not exist.
                File `/Users/sjors/dev/bitcoin/depends/aarch64-apple-darwin24.2.0/.native_libmultiprocess_stamp_configured' does not exist.
                 File `/Users/sjors/dev/bitcoin/depends/work/build/aarch64-apple-darwin24.2.0/native_libmultiprocess/-6dd32f3a9cb/.stamp_preprocessed' does not exist.
                  File `/Users/sjors/dev/bitcoin/depends/work/build/aarch64-apple-darwin24.2.0/native_libmultiprocess/-6dd32f3a9cb/.stamp_extracted' does not exist.
                   File `/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash' does not exist.
                  Must remake target `/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash'.
mkdir -p /Users/sjors/dev/bitcoin/depends/sources/download-stamps /Users/sjors/dev/bitcoin/depends/sources
rm -f /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
touch /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
cd /Users/sjors/dev/bitcoin/depends/sources/download-stamps; ( test -f /Users/sjors/dev/bitcoin/depends/sources/ || (     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from  && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- ) ||     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from https://bitcoincore.org/depends-sources && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "https://bitcoincore.org/depends-sources/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- )))
Fetching from
curl: (3) URL rejected: No host part in the URL
Fetching from https://bitcoincore.org/depends-sources
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403
make: *** [/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22

Hope that helps, happy to test more tomorrow.

[ryanofsky]

There's no --trace argument for make, did you mean --debug?

This might be a big hint. My version of make definitely has a --trace (GNU Make 4.4.1). Maybe the problem is caused by a difference in make versions? It is definitely a strange thing that the new code does not seem to be executed in your case. Would be curious to know your make --version and maybe I can experiment with different versions.

[Sjors]

% which make
/usr/bin/make
% make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

[Sjors]

It tried using gmake 4.4.1 installed via Homebrew, but that doesn't help. Not even with PATH="$HOMEBREW_PREFIX/opt/make/libexec/gnubin:$PATH".

It does however give me the --trace option and it echos HELLO from your patch, so here you go:

% which make
/opt/homebrew/opt/make/libexec/gnubin/make
% make --trace MULTIPROCESS=1 NO_QT=1 NO_ZMQ=1 NO_WALLET=1 NO_USDT=1 V=1
+ echo HELLO HELLO CREATING /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash
HELLO HELLO CREATING /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash
+ '[' -f /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash ']'
+ mkdir -p /Users/sjors/dev/bitcoin/depends/sources/download-stamps/
+ tar -c --sort=name --mtime=0 --owner=0 --group=0 --numeric-owner -f /Users/sjors/dev/bitcoin/depends/sources/src-ipc-libmultiprocess.tar -C ../src/ipc/libmultiprocess .
tar: Option --sort=name is not supported
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
+ cut '-d ' -f1 /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash
cut: /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash: No such file or directory
+ cat /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash
cat: /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-libmultiprocess-.hash: No such file or directory
+ echo HELLO HELLO CREATING /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
HELLO HELLO CREATING /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
+ '[' -f /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash ']'
+ mkdir -p /Users/sjors/dev/bitcoin/depends/sources/download-stamps/
+ tar -c --sort=name --mtime=0 --owner=0 --group=0 --numeric-owner -f /Users/sjors/dev/bitcoin/depends/sources/src-ipc-libmultiprocess.tar -C ../src/ipc/libmultiprocess .
tar: Option --sort=name is not supported
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
+ cut '-d ' -f1 /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
cut: /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash: No such file or directory
+ cat /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
cat: /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash: No such file or directory
Makefile:262: update target 'check-sources' due to: target is .PHONY

...

funcs.mk:318: update target '/Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash' due to: target does not exist
mkdir -p /Users/sjors/dev/bitcoin/depends/sources/download-stamps /Users/sjors/dev/bitcoin/depends/sources
rm -f /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
touch /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash
cd /Users/sjors/dev/bitcoin/depends/sources/download-stamps; ( test -f /Users/sjors/dev/bitcoin/depends/sources/ || (     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from  && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- ) ||     ( mkdir -p /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- && echo Fetching  from https://bitcoincore.org/depends-sources && curl --location --fail --connect-timeout 30 --retry 3 -o "/Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" "https://bitcoincore.org/depends-sources/" && echo "  /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp" > /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && shasum -a 256 -c /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/..hash && mv /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess-/.temp /Users/sjors/dev/bitcoin/depends/sources/ && rm -rf /Users/sjors/dev/bitcoin/depends/work/download/native_libmultiprocess- )))
Fetching from
curl: (3) URL rejected: No host part in the URL
Fetching from https://bitcoincore.org/depends-sources
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  308k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403
make: *** [funcs.mk:321: /Users/sjors/dev/bitcoin/depends/sources/download-stamps/.stamp_fetched-native_libmultiprocess-.hash] Error 22
sjors@Pzndsr4U depends % 

[Sjors]

% sort --version
2.3-Apple (190.0.1)

If I also add coreutils/libexec/gnubin to my path it still doesn't work, because tar doesn't use it.

% which tar
/usr/bin/tar

% tar --version
bsdtar 3.5.3 - libarchive 3.5.3 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 

Installing gnu-tar finally does the trick. But this seems a bit too invasive.

[ryanofsky]

Great! So there are two problems here. (1) The bigger problem is that the makefile code only seems to run in newer version of make to language differences and (2) the smaller problem is tar doesn't accept the same options.

The smaller problem (2) can be fixed by just dropping the options. They were added to make tarball generation more deterministic. But this is not actually important. The only thing that is important is that the build hash needs to change if the tarball contents change, it is fine if build hash changes even when tarball contents do not change, since tarball itself is meant to be just cached and used locally and only rebuilt if removed or the source is touched to have a newer mtime.

The problem (1) I will need to debug by testing with 3.81 locally. I'm not sure what the issue is but chatgpt had some ideas https://chatgpt.com/share/679a0d79-6e5c-800a-8285-be26d3d017e9

[ryanofsky]

If you want to try something else you could try chatgpt's suggestion of splitting the define int_get_local_dir into two parts: define int_get_local_dir and define do_local_dir_fetch which seems like it would be cleaner and could avoid some issues. Otherwise I will try 3.81 myself later today.

[Sjors]

I you can make me a patch I'll give it a try.

[hebasto]

(1) The bigger problem is that the makefile code only seems to run in newer version of make to language differences

This is not the first time when Apple's default make has caused problems.

Perhaps we should require a minimum version of GNU Make.

[Sjors]

Or switch depends to cmake?

I think it's fine to ask Apple users that want to build depends to brew install make and then use gmake instead of make. As long as we don't ask them to override the default make.

[hebasto]

Or switch depends to cmake?

I think it's fine to ask Apple users that want to build depends to brew install make and then use gmake instead of make. As long as we don't ask them to override the default make.

Btw, the gmake invocation works for GNU Make packages across multiple OSes, with the notable exception of Guix.