Merge #988: Make signing table fully static

7dfcece build: Remove #undef hack for ASM in the precomputation programs (Tim Ruffing)
bb36fe9 ci: Test `make precomp` (Tim Ruffing)
d94a37a build: Remove CC_FOR_BUILD stuff (Tim Ruffing)
ad63bb4 build: Prebuild and distribute ecmult_gen table (Tim Ruffing)
ac49361 prealloc: Get rid of manual memory management for prealloc contexts (Tim Ruffing)
6573c08 ecmult_gen: Tidy precomputed file and save space (Tim Ruffing)
5eba83f ecmult_gen: Precompute tables for all values of ECMULT_GEN_PREC_BITS (Tim Ruffing)
fdb33dd refactor: Make PREC_BITS a parameter of ecmult_gen_build_prec_table (Tim Ruffing)
a4875e3 refactor: Move default callbacks to util.h (Tim Ruffing)
4c94c55 doc: Remove obsolete hint for valgrind stack size (Tim Ruffing)
5106226 exhaustive_tests: Fix with ecmult_gen table with custom generator (Tim Ruffing)
e1a7653 refactor: Make generator a parameter of ecmult_gen_create_prec_table (Tim Ruffing)
9ad09f6 refactor: Rename program that generates static ecmult_gen table (Tim Ruffing)
8ae18f1 refactor: Rename file that contains static ecmult_gen table (Tim Ruffing)
00d2fa1 ecmult_gen: Make code consistent with comment (Tim Ruffing)
3b0c218 ecmult_gen: Simplify ecmult_gen context after making table static (Tim Ruffing)
e43ba02 refactor: Decouple table generation and ecmult_gen context (Tim Ruffing)
22dc2c0 ecmult_gen: Move table creation to new file and force static prec (Tim Ruffing)

Pull request description:

  This resolves #893,  resolves #692 (and also resolves bitcoin/bitcoin#22854).

  - [x] Extract table generation to separate function in separate file (to be used by generation script and exhaustive tests)
  - [x] Tidy up
    - [x] Remove code that deals with non-static tables
    - [x] Make functions that need ecmult_gen not depend on signing context
    - [x] Rename stuff to make it fit the new structure and consistent with how we hande verification tables (#956)
  - [x] Fix exhaustive tests
    - [x] Make table generation function take generator as input
    - [x] Overwrite the static tables with a table with custom generator in exhaustive tests
  - [x] Overhaul script that generates table files
    - [x] Make table generation function take PREC_BITS as input (I have some code already, just not yet in this branch)
    - [x] Change generation script to generate three tables (for all three values of ECMULT_GEN_PREC_BITS)
  - [x] Ship pre-built tables
    - [x] Add pregenerated table file to repo
    - [x] Remove generation of table file from build process (like in #956)
    - [x] Remove left-over stuff (e.g., detecting a compiler running on the build machine) from build system
  - [x] Final cleanups (copyright headers, commit, messages, etc.)
  - [ ] (separate PR:) Make sure link-time optimization remove corresponding static tables (and code) when no signing/verifcation function is called
  - [ ] (separate PR:) Compile precomputation as a separate object file and link it (#988 (comment))
  - [ ] (separate PR:) Document the backwards-compatible API changes made in this PR and in #956.
    - [ ] Maybe deprecate the static context

ACKs for top commit:
  sipa:
    ACK 7dfcece
  robot-dreams:
    ACK 7dfcece (based on range-diff between 56284c7 and 7dfcece)

Tree-SHA512: 6efb3f36f05efe3b79bbd877881fe1409f71fd6488d24c811b2e77d9f053bed78670dd1dcbb42ad780458a51c4ffa36de9cd6567271b22041dc7a122ceb677c5

Author: real-or-random

.cirrus.yml

@@ -294,7 +294,6 @@ task:
     - name: "UBSan, ASan, LSan"
       env:
         CFLAGS: "-fsanitize=undefined,address -g"
-        CFLAGS_FOR_BUILD: "-fsanitize=undefined,address -g"
         UBSAN_OPTIONS: "print_stacktrace=1:halt_on_error=1"
         ASAN_OPTIONS: "strict_string_checks=1:detect_stack_use_after_return=1:detect_leaks=1"
         LSAN_OPTIONS: "use_unaligned=1"

.gitattributes

@@ -1 +1,2 @@
 src/ecmult_static_pre_g.h linguist-generated
+src/ecmult_gen_static_prec_table.h linguist-generated

.gitignore

@@ -3,7 +3,7 @@ bench_ecmult
 bench_internal
 tests
 exhaustive_tests
-gen_context
+gen_ecmult_gen_static_prec_table
 gen_ecmult_static_pre_g
 valgrind_ctime_test
 *.exe
@@ -41,7 +41,6 @@ coverage.*.html
 
 src/libsecp256k1-config.h
 src/libsecp256k1-config.h.in
-src/ecmult_static_context.h
 build-aux/config.guess
 build-aux/config.sub
 build-aux/depcomp

Makefile.am

@@ -1,3 +1,5 @@
+.PHONY: clean-precomp precomp
+
 ACLOCAL_AMFLAGS = -I build-aux/m4
 
 # AM_CFLAGS will be automatically prepended to CFLAGS by Automake when compiling some foo
@@ -28,6 +30,8 @@ noinst_HEADERS += src/ecmult_const.h
 noinst_HEADERS += src/ecmult_const_impl.h
 noinst_HEADERS += src/ecmult_gen.h
 noinst_HEADERS += src/ecmult_gen_impl.h
+noinst_HEADERS += src/ecmult_gen_prec.h
+noinst_HEADERS += src/ecmult_gen_prec_impl.h
 noinst_HEADERS += src/field_10x26.h
 noinst_HEADERS += src/field_10x26_impl.h
 noinst_HEADERS += src/field_5x52.h
@@ -50,6 +54,7 @@ noinst_HEADERS += src/hash_impl.h
 noinst_HEADERS += src/field.h
 noinst_HEADERS += src/field_impl.h
 noinst_HEADERS += src/bench.h
+noinst_HEADERS += src/basic-config.h
 noinst_HEADERS += contrib/lax_der_parsing.h
 noinst_HEADERS += contrib/lax_der_parsing.c
 noinst_HEADERS += contrib/lax_der_privatekey_parsing.h
@@ -115,7 +120,7 @@ endif
 if USE_EXHAUSTIVE_TESTS
 noinst_PROGRAMS += exhaustive_tests
 exhaustive_tests_SOURCES = src/tests_exhaustive.c
-exhaustive_tests_CPPFLAGS = -I$(top_srcdir)/src $(SECP_INCLUDES)
+exhaustive_tests_CPPFLAGS = $(SECP_INCLUDES)
 if !ENABLE_COVERAGE
 exhaustive_tests_CPPFLAGS += -DVERIFY
 endif
@@ -124,36 +129,45 @@ exhaustive_tests_LDFLAGS = -static
 TESTS += exhaustive_tests
 endif
 
-EXTRA_PROGRAMS = gen_ecmult_static_pre_g
+### Precomputed tables
+EXTRA_PROGRAMS = gen_ecmult_static_pre_g gen_ecmult_gen_static_prec_table
+CLEANFILES = $(EXTRA_PROGRAMS)
+
 gen_ecmult_static_pre_g_SOURCES = src/gen_ecmult_static_pre_g.c
-# See Automake manual, Section "Errors with distclean"
+gen_ecmult_static_pre_g_CPPFLAGS = $(SECP_INCLUDES)
+gen_ecmult_static_pre_g_LDADD = $(SECP_LIBS) $(COMMON_LIB)
+
+gen_ecmult_gen_static_prec_table_SOURCES = src/gen_ecmult_gen_static_prec_table.c
+gen_ecmult_gen_static_prec_table_CPPFLAGS = $(SECP_INCLUDES)
+gen_ecmult_gen_static_prec_table_LDADD = $(SECP_LIBS) $(COMMON_LIB)
+
+# See Automake manual, Section "Errors with distclean".
+# We don't list any dependencies for the prebuilt files here because
+# otherwise make's decision whether to rebuild them (even in the first
+# build by a normal user) depends on mtimes, and thus is very fragile.
+# This means that rebuilds of the prebuilt files always need to be
+# forced by deleting them, e.g., by invoking `make clean-precomp`.
 src/ecmult_static_pre_g.h:
 	$(MAKE) $(AM_MAKEFLAGS) gen_ecmult_static_pre_g$(EXEEXT)
 	./gen_ecmult_static_pre_g$(EXEEXT)
+src/ecmult_gen_static_prec_table.h:
+	$(MAKE) $(AM_MAKEFLAGS) gen_ecmult_gen_static_prec_table$(EXEEXT)
+	./gen_ecmult_gen_static_prec_table$(EXEEXT)
 
-if USE_ECMULT_STATIC_PRECOMPUTATION
-CPPFLAGS_FOR_BUILD +=-I$(top_srcdir) -I$(builddir)/src
-
-gen_context_OBJECTS = gen_context.o
-gen_context_BIN = gen_context$(BUILD_EXEEXT)
-$(gen_context_OBJECTS): src/gen_context.c src/libsecp256k1-config.h
-	$(CC_FOR_BUILD) $(DEFS) $(CPPFLAGS_FOR_BUILD) $(SECP_CFLAGS_FOR_BUILD) $(CFLAGS_FOR_BUILD) -c $< -o $@
+PRECOMP = src/ecmult_gen_static_prec_table.h src/ecmult_static_pre_g.h
+noinst_HEADERS += $(PRECOMP)
+precomp: $(PRECOMP)
 
-$(gen_context_BIN): $(gen_context_OBJECTS)
-	$(CC_FOR_BUILD) $(SECP_CFLAGS_FOR_BUILD) $(CFLAGS_FOR_BUILD) $(LDFLAGS_FOR_BUILD) $^ -o $@
+# Ensure the prebuilt files will be build first (only if they don't exist,
+# e.g., after `make maintainer-clean`).
+BUILT_SOURCES = $(PRECOMP)
 
-$(libsecp256k1_la_OBJECTS): src/ecmult_static_context.h
-$(tests_OBJECTS): src/ecmult_static_context.h
-$(bench_internal_OBJECTS): src/ecmult_static_context.h
-$(bench_ecmult_OBJECTS): src/ecmult_static_context.h
+maintainer-clean-local: clean-precomp
 
-src/ecmult_static_context.h: $(gen_context_BIN)
-	./$(gen_context_BIN)
-
-CLEANFILES = $(gen_context_BIN) src/ecmult_static_context.h
-endif
+clean-precomp:
+	rm -f $(PRECOMP)
 
-EXTRA_DIST = autogen.sh src/gen_context.c src/ecmult_static_pre_g.h src/basic-config.h
+EXTRA_DIST = autogen.sh SECURITY.md
 
 if ENABLE_MODULE_ECDH
 include src/modules/ecdh/Makefile.am.include

README.md

@@ -66,18 +66,9 @@ libsecp256k1 is built using autotools:
     $ ./autogen.sh
     $ ./configure
     $ make
-    $ make check
+    $ make check  # run the test suite
     $ sudo make install  # optional
 
-Exhaustive tests
------------
-
-    $ ./exhaustive_tests
-
-With valgrind, you might need to increase the max stack size:
-
-    $ valgrind --max-stackframe=2500000 ./exhaustive_tests
-
 Test coverage
 -----------
 

build-aux/m4/ax_prog_cc_for_build.m4

@@ -48,7 +48,19 @@ then
         $EXEC ./bench
     } >> bench.log 2>&1
 fi
+
 if [ "$CTIMETEST" = "yes" ]
 then
     ./libtool --mode=execute valgrind --error-exitcode=42 ./valgrind_ctime_test > valgrind_ctime_test.log 2>&1
 fi
+
+# Rebuild precomputed files (if not cross-compiling).
+if [ -z "$HOST" ]
+then
+    make clean-precomp
+    make precomp
+fi
+
+# Check that no repo files have been modified by the build.
+# (This fails for example if the precomp files need to be updated in the repo.)
+git diff --exit-code