DRAFT: Upgrade to C++20

[drebelsky]

Upgrade to C++20. Notes:

• autocheck uses std::result_of (deprecated in 17, removed in 20)—I replaced just the use of that since we seem to have a de-facto in-tree fork of autocheck (e.g., we reference stellar::).

• fmt::format requires a either a compile-time constant first parameter in 20 or a fmt::runtime call.

• for fmt to work on clang, it seems that we need to have a version starting from this commit. Otherwise, code like the following doesn't work

  auto out = std::vector<char>();
  fmt::format_to(std::back_inserter(out), FMT_STRING("{}"), 42);

  So, I bumped the versions of fmt and spdlog

  • With the newer version of spdlog, FMT_STRING without any template arguments no longer picks up the right overload, but the C++20 version seems to correctly do compile-time strings with out it, so I updated Logging.h to reflect this. One alternative is using the following macro, but I think it's probably cleaner to just remove the FMT_STRING usage in Logging.h

  #define ID(ARG) ARG
  #define LPAREN (
  #define RPAREN )
  /* Expands to `FMT_STRING(f), __VA_ARGS__` when __VA_ARGS__ is non-empty and `f`
   * otherwise (spdlog will not pick the proper overload if we only pass
   * FMT_STRING(f)) */
  #define LOG_FMT_STRING(f, ...) \
      ID(__VA_OPT__(FMT_STRING LPAREN) f __VA_OPT__(RPAREN)) \
      __VA_OPT__(, ) __VA_ARGS__

• xdrpp needs to get updated for two reasons.

  1. On its current master, we end up trying to build using C++14, but it should be at least 17. Bumping to 20 (or 17) we get some extra flags¹

     checking whether ccache clang++ -std=c++20 accepts -g... yes
     checking for ccache clang++ -std=c++20 option to enable C++11 features... -std=gnu++11
     checking dependency style of ccache clang++ -std=c++20 -std=gnu++11... gcc3
     checking whether ccache clang++ -std=c++20 -std=gnu++11 supports C++20 features by default... no
     checking whether ccache clang++ -std=c++20 -std=gnu++11 supports C++20 features with -std=gnu++20... yes
     

  2. Because of the spaceship operator, at the very least, we need operator<=> defined for xdr::pointer since it inherits from std::unique_ptr which defines <=>, meaning that without the overload, e.g., std::vector<xdr::pointer<int>> comparisons no longer work properly (they go back to comparing by address instead of value).

  There is the cxx20 branch, but it doesn't have the most recent 5 commits from the master branch (which we rely on in core). Additionally, it seems to be somewhat buggy (e.g., tests/marshal.cc doesn't pass). So, for the purposes of this PR, I think it's probably easiest to just bump to 20 and add operator<=> for xdr data types.

• Small change, but renamed the constructor in the header file for ScopedLedgerEntry since ScopedLedgerEntry<T> isn't technically valid since DR 2237 was adopted in C++20 (link).

Other notes on C++20 compatibility. I compiled with -Wc++20-compat locally, and I didn't see any warnings. (Maybe we should do this in CI too?) Looking at C.1 diff.cpp17, on a version of this PR based on 414a5e5,

Checked

• C.1.2.3, C.1.2.5, C.1.5.4, C.1.6.1, C.1.6.2, C.1.14.1, C.1.14.4, C.1.15: The "effect on original feature" for all of these say something to the effect of "Valid C++ 2017 code... is not valid in this revision of C++" (or "may fail to compile"). Since we compile on 20, these are all okay.

• C.1.5.1, C.1.8.1, C.1.9.1, C.1.14.2, C.1.14.3: The "effect on original feature" for all of these say something to the effect of "Valid C++ 2017 code may be ill-formed in this revision of C++" (n.b., not "Valid C++ 2017 code may be ill-formed in this revision of C++, with no diagnostic required."). Since we compile on 20, these are all okay.

• C.1.2.1: "Effect on original feature: Logical lines beginning with module or import may be interpreted differently in this revision of C++." The only lines we have that begin with "import" or "module" are in the updated fmt library

  Click to expand

  $ rg -uuu -g "*.cpp" -g "*.cc" -g "*.cxx" -g "*.C" -g "*.c++" -g "*.hpp" \
            -g "*.hh" -g "*.h" -g "*.hxx" -g "*.H" -g "*.h++" -g "*.ipp" \
            -g "*.tpp" -g "*.txx" \
        "^(import|module)"
  lib/fmt/src/fmt.cc
  1:module;
  97:import std;
  139:module :private;
  
  lib/fmt/test/module-test.cc
  43:import fmt;
  

  (note that afaict, our codebase only has the following file endings, the others are just for posterity: .cc, .cpp, .h, .hh, .hpp, .ipp)

• C.1.2.2: "Effect on original feature: When the identifier import is followed by a < character, a header-name token may be formed." These only appear in Objective-C code that we don't actually compile (in catch.hpp, they're in #ifdef __OBJC__ blocks and the gtest is an xcode subdirectory

  Click to expand

  $ rg -uuu -g "*.cpp" -g "*.cc" -g "*.cxx" -g "*.C" -g "*.c++" -g "*.hpp" \
            -g "*.hh" -g "*.h" -g "*.hxx" -g "*.H" -g "*.h++" -g "*.ipp" \
            -g "*.tpp" -g "*.txx" \
       "import\s*<"
  lib/catch.hpp
  1503:#import <Foundation/Foundation.h>
  4851:#import <objc/runtime.h>
  
  lib/libmedida/test/gtest-1.6.0/xcode/Samples/FrameworkSample/widget.h
  40:#import <string>
  

• C.1.2.4: "Effect on original feature: Valid C++ 2017 code that contains a <= token immediately followed by a > token may be ill-formed or have different semantics in this revision of C++." All instances are either in our new 20 code, a comment, or a string literal

  Click to expand

  $ rg -uuu -g "*.cpp" -g "*.cc" -g "*.cxx" -g "*.C" -g "*.c++" -g "*.hpp" \
            -g "*.hh" -g "*.h" -g "*.hxx" -g "*.H" -g "*.h++" -g "*.ipp" \
            -g "*.tpp" -g "*.txx" \
       "<=>"
  lib/cereal/include/cereal/external/rapidjson/writer.h
  608:        const __m128i t3 = _mm_cmpeq_epi8(_mm_max_epu8(s, sp), sp); // s < 0x20 
  <=> max(s, 0x1F) == 0x1F
  
  lib/cereal/include/cereal/external/rapidjson/document.h
  189:    template <bool Const_> std::strong_ordering operator<=>(const GenericMemberI
  terator<Const_, Encoding, Allocator>& that) const { return ptr_ <=> that.ptr_; }
  
  lib/cereal/include/cereal/external/rapidjson/reader.h
  1088:            const __m128i t3 = _mm_cmpeq_epi8(_mm_max_epu8(s, sp), sp); // s < 
  0x20 <=> max(s, 0x1F) == 0x1F
  1151:            const __m128i t3 = _mm_cmpeq_epi8(_mm_max_epu8(s, sp), sp); // s < 
  0x20 <=> max(s, 0x1F) == 0x1F
  1199:            const __m128i t3 = _mm_cmpeq_epi8(_mm_max_epu8(s, sp), sp); // s < 
  0x20 <=> max(s, 0x1F) == 0x1F
  1638:                    //   <=>  exp <= (expFrac - INT_MIN - 9) / 10
  
  lib/fmt/test/gtest/gmock-gtest-all.cc
  11111:  return IsInSet(ch, "^-!\"#$%&'()*+,./:;<=>?@[\\]_`{|}~");
  
  lib/tracy/import-chrome/src/json.hpp
  2884:    inline std::partial_ordering operator<=>(const value_t lhs, const value_t r
  hs) noexcept // *NOPAD*
  2901:        return order[l_index] <=> order[r_index]; // *NOPAD*
  2916:    return std::is_lt(lhs <=> rhs); // *NOPAD*
  14550:    std::strong_ordering operator<=>(const json_pointer<RefStringTypeRhs>& rhs
  ) const noexcept // *NOPAD*
  14552:        return  reference_tokens <=> rhs.reference_tokens; // *NOPAD*
  22887:    std::partial_ordering operator<=>(const_reference rhs) const noexcept // *
  NOPAD*
  22892:        JSON_IMPLEMENT_OPERATOR(<=>, // *NOPAD*
  22895:                                lhs_type <=> rhs_type) // *NOPAD*
  22902:    std::partial_ordering operator<=>(ScalarType rhs) const noexcept // *NOPAD
  *
  22904:        return *this <=> basic_json(rhs); // *NOPAD*
  24361:        return std::is_lt(lhs <=> rhs); // *NOPAD*
  
  lib/libmedida/test/gtest-1.6.0/src/gtest-port.cc
  187:  return IsInSet(ch, "^-!\"#$%&'()*+,./:;<=>?@[\\]_`{|}~");
  
  lib/libmedida/test/gtest-1.6.0/fused-src/gtest/gtest-all.cc
  7989:  return IsInSet(ch, "^-!\"#$%&'()*+,./:;<=>?@[\\]_`{|}~");
  
  lib/xdrpp/xdrpp/types.h
  634:  friend std::partial_ordering operator<=>(const pointer &a, const pointer &b) {
  639:    return *a <=> *b;
  971:operator<=>(const T &a, const T &b)
  991:    out = a.*mp <=> b.*mp;
  1015:operator<=>(const T &a, const T &b)
  1018:      a._xdr_discriminant() <=> b._xdr_discriminant();
  1033:    if (std::partial_ordering res = fi(a) <=> fi(b);
  
  lib/libmedida/test/gtest-1.6.0/test/gtest-port_test.cc
  468:  for (const char* p = "^-!\"#$%&'()*+,./:;<=>?@[\\]_`{|}~"; *p; p++) {
  
  src/util/XDROperators.h
  12:using xdr::operator<=>;
  

• C.1.10: we compile, so these should be okay

Not (fully) checked

• C.1.3: it seems pretty unlikely that we would call a pseudo-destructor, and I believe most of the memory ordering usage happens in tracy, so it's not terrible that there's this slight semantic change
• C.1.4. I could only imagine this being problematic from a safety perspective if this ends up with us having dangling references that we didn't have in 17, but since the difference only applies to things that aren't odr-used, I believe we should be fine.
• C.1.5.2, C.1.5.3, C.1.6.3, C.1.7
• C.1.11: "Effect on original feature: Code that depends on the return types might have different semantics in this revision of C++. Translation units compiled against this version of C++ may be incompatible with translation units compiled against C++ 2017, either failing to link or having undefined behavior." We compile all our TUs with C++20. I haven't checked every instance, but it seems unlikely that we would rely on std::{list,forward_list}::{remove,remove_if,unique} returning void (instead of container::size_t).
• C.1.12, C.1.13

Footnotes

1. Due to a known issue: autoconf adds a flag to enable C++11 features since C++20 isn't fully backwards compatible. Unfortunately, the patch to fix this behavior is not in a released version of autoconf ↩

[drebelsky]

Looking at it now, I see we had some custom overrides in autocheck (e.g., using stellar:: prefixed random number generators), so I will re-evaluate that (maybe we should fork autocheck?).

[drebelsky]

Looks like this failed last time because of xdrpp building on C++14, bumping to the commit introduced in this pr

[drebelsky]

The test case "LedgerTxnRoot prefetch classic entries" fails because vectors of xdr::pointers break (due to the spaceship operator overloads) in C++20, so I will work on merging the cxx20 branch with master in xdrpp.

Edit: cxx20 branch seems to have some problems, updated xdrpp to build on 20

[drebelsky]

I think these failures are related to the tcmalloc issue

stellar-core: error while loading shared libraries: libtcmalloc_minimal.so.4: cannot open shared object file: No such file or directory