Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmake: Add Coverage script #191

Merged
merged 1 commit into from
Jun 8, 2024
Merged

cmake: Add Coverage script #191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,22 @@ else()
unset(c_flags_debug_overridden)
endif()

set(CMAKE_CXX_FLAGS_COVERAGE "-Og --coverage")
set(CMAKE_OBJCXX_FLAGS_COVERAGE "-Og --coverage")
set(CMAKE_EXE_LINKER_FLAGS_COVERAGE "--coverage")
set(CMAKE_SHARED_LINKER_FLAGS_COVERAGE "--coverage")
get_property(is_multi_config GLOBAL PROPERTY GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
if(NOT "Coverage" IN_LIST CMAKE_CONFIGURATION_TYPES)
list(APPEND CMAKE_CONFIGURATION_TYPES Coverage)
endif()
endif()

configure_file(cmake/script/Coverage.cmake Coverage.cmake COPYONLY)
configure_file(cmake/script/CoverageFuzz.cmake CoverageFuzz.cmake COPYONLY)
configure_file(cmake/script/CoverageInclude.cmake.in CoverageInclude.cmake @ONLY)
configure_file(contrib/filter-lcov.py filter-lcov.py COPYONLY)

include(cmake/optional.cmake)

# Don't allow extended (non-ASCII) symbols in identifiers. This is easier for code review.
Expand Down
5 changes: 5 additions & 0 deletions cmake/cov_tool_wrapper.sh.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Copyright (c) 2024-present The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or https://opensource.org/license/mit/.

exec @COV_TOOL@ "$@"
77 changes: 77 additions & 0 deletions cmake/script/Coverage.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Copyright (c) 2024-present The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or https://opensource.org/license/mit/.

include(${CMAKE_CURRENT_LIST_DIR}/CoverageInclude.cmake)

set(functional_test_runner test/functional/test_runner.py)
if(EXTENDED_FUNCTIONAL_TESTS)
list(APPEND functional_test_runner --extended)
endif()
if(DEFINED JOBS)
list(APPEND CMAKE_CTEST_COMMAND -j ${JOBS})
list(APPEND functional_test_runner -j ${JOBS})
endif()

execute_process(
COMMAND ${CMAKE_CTEST_COMMAND} --build-config Coverage
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
COMMAND_ERROR_IS_FATAL ANY
)
execute_process(
COMMAND ${LCOV_COMMAND} --capture --directory src --test-name test_bitcoin --output-file test_bitcoin.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --zerocounters --directory src
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_FILTER_COMMAND} test_bitcoin.info test_bitcoin_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
Comment on lines +29 to +32
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file consists mostly of such execute_process() calls. The most natural way to execute a bunch of commands is from a shell script. I would suggest to use a shell script for this. This 77 lines cmake file would be equivalent roughly to the following:

functional_test_runner="test/functional/test_runner.py"
if [ -n "$EXTENDED_FUNCTIONAL_TESTS" ] ; then
    functional_test_runner="$functional_test_runner --extended"
fi
if [ -n "$JOBS" ] ; then
    CMAKE_CTEST_COMMAND="$CMAKE_CTEST_COMMAND -j $JOBS"
    functional_test_runner="$functional_test_runner -j $JOBS"
fi

cd $CMAKE_CURRENT_LIST_DIR
$CMAKE_CTEST_COMMAND --build-config Coverage
$LCOV_COMMAND --capture --directory src --test-name test_bitcoin --output-file test_bitcoin.info
$LCOV_COMMAND --zerocounters --directory src
$LCOV_FILTER_COMMAND test_bitcoin.info test_bitcoin_filtered.info
$LCOV_COMMAND --add-tracefile test_bitcoin_filtered.info --output-file test_bitcoin_filtered.info
$LCOV_COMMAND --add-tracefile baseline_filtered.info --add-tracefile test_bitcoin_filtered.info --output-file test_bitcoin_coverage.info
$GENHTML_COMMAND test_bitcoin_coverage.info --output-directory test_bitcoin.coverage
$functional_test_runner
$LCOV_COMMAND --capture --directory src --test-name functional-tests --output-file functional_test.info
$LCOV_COMMAND --zerocounters --directory src
$LCOV_FILTER_COMMAND functional_test.info functional_test_filtered.info
$LCOV_COMMAND --add-tracefile functional_test_filtered.info --output-file functional_test_filtered.info
$LCOV_COMMAND --add-tracefile baseline_filtered.info --add-tracefile test_bitcoin_filtered.info --add-tracefile functional_test_filtered.info --output-file total_coverage.info | $GREP_EXECUTABLE "%" | $AWK_EXECUTABLE "{ print substr($3,2,50) \"/\" $5 }" > coverage_percent.txt
$GENHTML_COMMAND total_coverage.info --output-directory total.coverage

Copy link
Owner Author

@hebasto hebasto May 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most natural way to execute a bunch of commands is from a shell script. I would suggest to use a shell script for this.

From CMake's point of view, using a platform independent approach is more natural :)

However, I have to admit that it is not expected that the user will run coverage scripts on Windows.

execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile test_bitcoin_filtered.info --output-file test_bitcoin_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile baseline_filtered.info --add-tracefile test_bitcoin_filtered.info --output-file test_bitcoin_coverage.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${GENHTML_COMMAND} test_bitcoin_coverage.info --output-directory test_bitcoin.coverage
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)

execute_process(
COMMAND ${functional_test_runner}
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
COMMAND_ERROR_IS_FATAL ANY
)
execute_process(
COMMAND ${LCOV_COMMAND} --capture --directory src --test-name functional-tests --output-file functional_test.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --zerocounters --directory src
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_FILTER_COMMAND} functional_test.info functional_test_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile functional_test_filtered.info --output-file functional_test_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile baseline_filtered.info --add-tracefile test_bitcoin_filtered.info --add-tracefile functional_test_filtered.info --output-file total_coverage.info
COMMAND ${GREP_EXECUTABLE} "%"
COMMAND ${AWK_EXECUTABLE} "{ print substr($3,2,50) \"/\" $5 }"
OUTPUT_FILE coverage_percent.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${GENHTML_COMMAND} total_coverage.info --output-directory total.coverage
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
42 changes: 42 additions & 0 deletions cmake/script/CoverageFuzz.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Copyright (c) 2024-present The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or https://opensource.org/license/mit/.

include(${CMAKE_CURRENT_LIST_DIR}/CoverageInclude.cmake)

if(NOT DEFINED FUZZ_SEED_CORPUS_DIR)
set(FUZZ_SEED_CORPUS_DIR ${CMAKE_CURRENT_SOURCE_DIR}/qa-assets/fuzz_seed_corpus)
endif()

execute_process(
COMMAND test/fuzz/test_runner.py ${FUZZ_SEED_CORPUS_DIR} --loglevel DEBUG
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
COMMAND_ERROR_IS_FATAL ANY
)
execute_process(
COMMAND ${LCOV_COMMAND} --capture --directory src --test-name fuzz-tests --output-file fuzz.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --zerocounters --directory src
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_FILTER_COMMAND} fuzz.info fuzz_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile fuzz_filtered.info --output-file fuzz_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile baseline_filtered.info --add-tracefile fuzz_filtered.info --output-file fuzz_coverage.info
COMMAND ${GREP_EXECUTABLE} "%"
COMMAND ${AWK_EXECUTABLE} "{ print substr($3,2,50) \"/\" $5 }"
OUTPUT_FILE coverage_percent.txt
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${GENHTML_COMMAND} fuzz_coverage.info --output-directory fuzz.coverage
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
56 changes: 56 additions & 0 deletions cmake/script/CoverageInclude.cmake.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Copyright (c) 2024-present The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or https://opensource.org/license/mit/.

if("@CMAKE_CXX_COMPILER_ID@" STREQUAL "Clang")
find_program(LLVM_COV_EXECUTABLE llvm-cov REQUIRED)
set(COV_TOOL "${LLVM_COV_EXECUTABLE} gcov")
Comment on lines +5 to +7
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clang has a different native workflow: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html which I guess produces better results compared to its gcc compatibility layer.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. But this PR aims to mirror the master branch behaviour. Further improvements can be done later, no?

else()
find_program(GCOV_EXECUTABLE gcov REQUIRED)
set(COV_TOOL "${GCOV_EXECUTABLE}")
endif()

# COV_TOOL is used to replace a placeholder.
configure_file(
cmake/cov_tool_wrapper.sh.in ${CMAKE_CURRENT_LIST_DIR}/cov_tool_wrapper.sh
FILE_PERMISSIONS OWNER_READ OWNER_EXECUTE
GROUP_READ GROUP_EXECUTE
WORLD_READ
@ONLY
)

find_program(LCOV_EXECUTABLE lcov REQUIRED)
separate_arguments(LCOV_OPTS)
set(LCOV_COMMAND ${LCOV_EXECUTABLE} --gcov-tool ${CMAKE_CURRENT_LIST_DIR}/cov_tool_wrapper.sh ${LCOV_OPTS})

find_program(GENHTML_EXECUTABLE genhtml REQUIRED)
set(GENHTML_COMMAND ${GENHTML_EXECUTABLE} --show-details ${LCOV_OPTS})

find_program(GREP_EXECUTABLE grep REQUIRED)
find_program(AWK_EXECUTABLE awk REQUIRED)

set(LCOV_FILTER_COMMAND ./filter-lcov.py)
list(APPEND LCOV_FILTER_COMMAND -p "/usr/local/")
list(APPEND LCOV_FILTER_COMMAND -p "/usr/include/")
list(APPEND LCOV_FILTER_COMMAND -p "/usr/lib/")
list(APPEND LCOV_FILTER_COMMAND -p "/usr/lib64/")
list(APPEND LCOV_FILTER_COMMAND -p "src/leveldb/")
list(APPEND LCOV_FILTER_COMMAND -p "src/crc32c/")
list(APPEND LCOV_FILTER_COMMAND -p "src/bench/")
list(APPEND LCOV_FILTER_COMMAND -p "src/crypto/ctaes")
list(APPEND LCOV_FILTER_COMMAND -p "src/minisketch")
list(APPEND LCOV_FILTER_COMMAND -p "src/secp256k1")
list(APPEND LCOV_FILTER_COMMAND -p "depends")

execute_process(
COMMAND ${LCOV_COMMAND} --capture --initial --directory src --output-file baseline.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_FILTER_COMMAND} baseline.info baseline_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
execute_process(
COMMAND ${LCOV_COMMAND} --add-tracefile baseline_filtered.info --output-file baseline_filtered.info
WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR}
)
23 changes: 16 additions & 7 deletions doc/developer-notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -481,25 +481,34 @@ $ ./test/functional/test_runner.py --valgrind

### Compiling for test coverage

LCOV can be used to generate a test coverage report based upon `make check`
LCOV can be used to generate a test coverage report based upon `ctest`
execution. LCOV must be installed on your system (e.g. the `lcov` package
on Debian/Ubuntu).

To enable LCOV report generation during test runs:

```shell
./configure --enable-lcov
make
make cov
cmake -B build -DCMAKE_BUILD_TYPE=Coverage
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think with/without coverage should be independent option from debug/release. For example one may want to gather coverage on debug builds or release builds? I do not think we need a dedicated build type "Coverage" in order to pass a few extra flags which is already supported by the build system with APPEND_..._FLAGS.

For clang, that would be extra -fprofile-instr-generate -fcoverage-mapping and compiling and running the tests as usual. I guess for gcc that would be --coverage instead.

I know this would be different from how autotools works, but I would propose to have a shell script in contrib/ that compiles with coverage enabled (e.g. by running cmake -DAPPEND_..._FLAGS="-fprofile-instr-generate -fcoverage-mapping"), then runs the tests and gathers the coverage (actually 2 shell scripts - one for clang and one for gcc).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example one may want to gather coverage on debug builds or release builds?

Is this something that is commonly done? What would the purpose of this be?

I know this would be different from how autotools works, but I would propose to have a shell script

Not sure about this. I prefer to work with the build system directly without having to understand a wrapper script first if I want to tweak something.

cmake --build build
cmake -P build/Coverage.cmake

# A coverage report will now be accessible at `./test_bitcoin.coverage/index.html`,
# which covers unit tests, and `./total.coverage/index.html`, which covers
# A coverage report will now be accessible at `./build/test_bitcoin.coverage/index.html`,
# which covers unit tests, and `./build/total.coverage/index.html`, which covers
# unit and functional tests.
```

Additional LCOV options can be specified using `LCOV_OPTS`, but may be dependant
on the version of LCOV. For example, when using LCOV `2.x`, branch coverage can be
enabled by setting `LCOV_OPTS="--rc branch_coverage=1"`, when configuring.
enabled by setting `LCOV_OPTS="--rc branch_coverage=1"`:

```
cmake -DLCOV_OPTS="--rc branch_coverage=1" -P build/Coverage.cmake
```

To enable test parallelism:
```
cmake -DJOBS=$(nproc) -P build/Coverage.cmake
```

### Performance profiling with perf

Expand Down
Loading