l2switch example fails to link with p4rt option

[msg-programs]

While working in the same Ubuntu 21.04 VM as in Issue #41 , I've encountered another problem. This time, I've tried to compile the program with P4Runtime support like so:

./t4p4s.sh :l2switch dbg verbose model=psa p4rt

Now, the compilation fails with a linker error. Here is the output generated (minor formatting added):

root@p4:/home/p4/Schreibtisch/t4p4s/t4p4s# ./t4p4s.sh :l2switch dbg verbose model=psa p4rt

Parsing variant config file examples.cfg
Parsing file opts_dpdk.cfg
Parse port is 51973
P4 version is 16 (by the extension of ./examples/l2switch.p4 )
Using CC=clang-12, LD=lld-12, PYTHON3=python3.9, DBG=gdb

Options: ^silent,arch=dpdk,cfgfiles,cflags=-D__TARGET_PSA__ -DT4P4S_DEBUG=on -DT4P4S_P4RT,choice=l2switch@std,example=l2switch,executable=./build/last/build/l2switch,hugemb=4096,include-hdrs=dpdk_nicon.h gen_model.h,include-srcs=hdep/'dpdk/dpdk_nicon.c' xtrn/'dpdk_model_psa.c' xtrn/'dpdkx_psa_checksum.c' xtrn/'dpdkx_psa_clone.c' xtrn/'dpdkx_psa_digest.c' xtrn/'dpdkx_psa_hash.c' xtrn/'dpdkx_psa_random.c' xtrn/'dpdkx_psa_smem.c' xtrn/'dpdkx_psa_digest.c',light=green,yellow,cyan,red,meson_opts,model=psa,p4opts=--p4opt=__TARGET_PSA__ --p4opt=USE_PSA,source=./examples/l2switch.p4,variant=std,vsn=16,c,dbg,l2switch,p4,p4rt,run,smem,verbose,

Requesting root access...
Root access granted, starting...
[NO  CONTROLLER]
[COMPILE  P4-16] ./examples/l2switch.p4 @std, debug mode, model psa
P4 compiler options : --p4opt=__TARGET_PSA__ --p4opt=USE_PSA ./examples/l2switch.p4 --p4v 16 -g ./build/last/srcgen -verbose 
HLIR (cached: stage hlir_add_attributes) build/last/cache/l2switch.p4.hlir.attributed.cached
[COMPILE SWITCH]
C compiler options: -D__TARGET_PSA__ -DT4P4S_DEBUG=on -DT4P4S_P4RT 

[1/56] Compiling C object l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_messages.c.o
clang: warning: argument unused during compilation: '-L/home/p4/Schreibtisch/t4p4s/P4Runtime_GRPCPP/static_lib' [-Wunused-command-line-argument] 
...
// this error repeats for every step up to [55/56]
...
[56/56] Linking target l2switch

FAILED: l2switch 
clang-12  -o l2switch l2switch.p/srcgen_dataplane.c.o l2switch.p/srcgen_dataplane_deparse.c.o l2switch.p/srcgen_dataplane_key.c.o l2switch.p/srcgen_dataplane_show.c.o l2switch.p/srcgen_dataplane_smem.c.o l2switch.p/srcgen_dataplane_stats.c.o l2switch.p/srcgen_dataplane_table_default_entry.c.o l2switch.p/srcgen_dataplane_update.c.o l2switch.p/srcgen_tables.c.o l2switch.p/srcgen_parser.c.o l2switch.p/srcgen_hdr_fld.c.o l2switch.p/srcgen_actions.c.o l2switch.p/srcgen_controlplane.c.o l2switch.p/srcgen_stats.c.o l2switch.p/srcgen_stats_requirements.c.o l2switch.p/srcgen_common.c.o l2switch.p/.._.._src_hardware_dep_dpdk_main.c.o l2switch.p/.._.._src_hardware_dep_dpdk_main_async.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_ctrl_plane_backend.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_fifo.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_handlers.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_messages.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_sock_helpers.c.o l2switch.p/.._.._src_hardware_dep_shared_ctrl_plane_threadpool.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_dpdk_lib.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_dpdk_tables.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_dpdk_primitives.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_dpdk_primitives_impl.c.o l2switch.p/.._.._src_hardware_dep_shared_data_plane_ternary_naive.c.o l2switch.p/srcgen_dpdkx_gen_extern.c.o l2switch.p/.._.._src_hardware_dep_shared_util_debug.c.o l2switch.p/.._.._src_hardware_dep_shared_backend.c.o l2switch.p/.._.._src_hardware_dep_shared_dataplane_hdr_fld_pkt.c.o l2switch.p/srcgen_stateful_memory.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_smem.c.o l2switch.p/.._.._src_hardware_dep_dpdk_dpdk_nicon.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdk_model_psa.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_checksum.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_clone.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_digest.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_hash.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_random.c.o l2switch.p/.._.._src_hardware_dep_dpdk_data_plane_externs_dpdkx_psa_smem.c.o l2switch.p/srcgen_multi_actions.stage_0.c.o l2switch.p/srcgen_multi_controlplane.stage_0.c.o l2switch.p/srcgen_multi_dataplane.stage_0.c.o l2switch.p/srcgen_multi_parser.stage_0.c.o l2switch.p/srcgen_multi_actions.stage_1.c.o l2switch.p/srcgen_multi_controlplane.stage_1.c.o l2switch.p/srcgen_multi_dataplane.stage_1.c.o l2switch.p/srcgen_multi_actions.stage_2.c.o l2switch.p/srcgen_multi_controlplane.stage_2.c.o l2switch.p/srcgen_multi_dataplane.stage_2.c.o l2switch.p/srcgen_multi_actions.stage_3.c.o l2switch.p/srcgen_multi_dataplane.stage_3.c.o -Wl,--as-needed -Wl,--no-undefined -fuse-ld=lld-12 -Wl,-rpath,/usr/local/lib/x86_64-linux-gnu:/usr/local/lib -Wl,-rpath-link,/usr/local/lib/x86_64-linux-gnu -Wl,-rpath-link,/usr/local/lib -Wl,--start-group /usr/local/lib/x86_64-linux-gnu/librte_bus_vdev.so -Wl,--as-needed /usr/local/lib/x86_64-linux-gnu/librte_node.so /usr/local/lib/x86_64-linux-gnu/librte_graph.so /usr/local/lib/x86_64-linux-gnu/librte_flow_classify.so /usr/local/lib/x86_64-linux-gnu/librte_pipeline.so /usr/local/lib/x86_64-linux-gnu/librte_table.so /usr/local/lib/x86_64-linux-gnu/librte_pdump.so /usr/local/lib/x86_64-linux-gnu/librte_port.so /usr/local/lib/x86_64-linux-gnu/librte_fib.so /usr/local/lib/x86_64-linux-gnu/librte_ipsec.so /usr/local/lib/x86_64-linux-gnu/librte_vhost.so /usr/local/lib/x86_64-linux-gnu/librte_stack.so /usr/local/lib/x86_64-linux-gnu/librte_security.so /usr/local/lib/x86_64-linux-gnu/librte_sched.so /usr/local/lib/x86_64-linux-gnu/librte_reorder.so /usr/local/lib/x86_64-linux-gnu/librte_rib.so /usr/local/lib/x86_64-linux-gnu/librte_dmadev.so /usr/local/lib/x86_64-linux-gnu/librte_regexdev.so /usr/local/lib/x86_64-linux-gnu/librte_rawdev.so /usr/local/lib/x86_64-linux-gnu/librte_power.so /usr/local/lib/x86_64-linux-gnu/librte_pcapng.so /usr/local/lib/x86_64-linux-gnu/librte_member.so /usr/local/lib/x86_64-linux-gnu/librte_lpm.so /usr/local/lib/x86_64-linux-gnu/librte_latencystats.so /usr/local/lib/x86_64-linux-gnu/librte_kni.so /usr/local/lib/x86_64-linux-gnu/librte_jobstats.so /usr/local/lib/x86_64-linux-gnu/librte_ip_frag.so /usr/local/lib/x86_64-linux-gnu/librte_gso.so /usr/local/lib/x86_64-linux-gnu/librte_gro.so /usr/local/lib/x86_64-linux-gnu/librte_gpudev.so /usr/local/lib/x86_64-linux-gnu/librte_eventdev.so /usr/local/lib/x86_64-linux-gnu/librte_efd.so /usr/local/lib/x86_64-linux-gnu/librte_distributor.so /usr/local/lib/x86_64-linux-gnu/librte_cryptodev.so /usr/local/lib/x86_64-linux-gnu/librte_compressdev.so /usr/local/lib/x86_64-linux-gnu/librte_cfgfile.so /usr/local/lib/x86_64-linux-gnu/librte_bpf.so /usr/local/lib/x86_64-linux-gnu/librte_bitratestats.so /usr/local/lib/x86_64-linux-gnu/librte_bbdev.so /usr/local/lib/x86_64-linux-gnu/librte_acl.so /usr/local/lib/x86_64-linux-gnu/librte_timer.so /usr/local/lib/x86_64-linux-gnu/librte_hash.so /usr/local/lib/x86_64-linux-gnu/librte_metrics.so /usr/local/lib/x86_64-linux-gnu/librte_cmdline.so /usr/local/lib/x86_64-linux-gnu/librte_pci.so /usr/local/lib/x86_64-linux-gnu/librte_ethdev.so /usr/local/lib/x86_64-linux-gnu/librte_meter.so /usr/local/lib/x86_64-linux-gnu/librte_net.so /usr/local/lib/x86_64-linux-gnu/librte_mbuf.so /usr/local/lib/x86_64-linux-gnu/librte_mempool.so /usr/local/lib/x86_64-linux-gnu/librte_rcu.so /usr/local/lib/x86_64-linux-gnu/librte_ring.so /usr/local/lib/x86_64-linux-gnu/librte_eal.so /usr/local/lib/x86_64-linux-gnu/librte_telemetry.so /usr/local/lib/x86_64-linux-gnu/librte_kvargs.so -pthread /usr/local/lib/libgrpc++.a /usr/local/lib/libgrpc.a /usr/local/lib/libaddress_sorting.a /usr/local/lib/libre2.a /usr/local/lib/libupb.a /usr/local/lib/libcares.a /usr/local/lib/libz.so /usr/local/lib/libgpr.a /usr/local/lib/libssl.a /usr/local/lib/libcrypto.a /usr/local/lib/libabsl_raw_hash_set.a /usr/local/lib/libabsl_hashtablez_sampler.a /usr/local/lib/libabsl_hash.a /usr/local/lib/libabsl_city.a /usr/local/lib/libabsl_low_level_hash.a /usr/local/lib/libabsl_random_distributions.a /usr/local/lib/libabsl_random_seed_sequences.a /usr/local/lib/libabsl_random_internal_pool_urbg.a /usr/local/lib/libabsl_random_internal_randen.a /usr/local/lib/libabsl_random_internal_randen_hwaes.a /usr/local/lib/libabsl_random_internal_randen_hwaes_impl.a /usr/local/lib/libabsl_random_internal_randen_slow.a /usr/local/lib/libabsl_random_internal_platform.a /usr/local/lib/libabsl_random_internal_seed_material.a /usr/local/lib/libabsl_random_seed_gen_exception.a /usr/local/lib/libabsl_statusor.a /usr/local/lib/libabsl_status.a /usr/local/lib/libabsl_cord.a /usr/local/lib/libabsl_cordz_info.a /usr/local/lib/libabsl_cord_internal.a /usr/local/lib/libabsl_cordz_functions.a /usr/local/lib/libabsl_exponential_biased.a /usr/local/lib/libabsl_cordz_handle.a /usr/local/lib/libabsl_bad_optional_access.a /usr/local/lib/libabsl_str_format_internal.a /usr/local/lib/libabsl_synchronization.a /usr/local/lib/libabsl_graphcycles_internal.a /usr/local/lib/libabsl_stacktrace.a /usr/local/lib/libabsl_symbolize.a /usr/local/lib/libabsl_debugging_internal.a /usr/local/lib/libabsl_demangle_internal.a /usr/local/lib/libabsl_malloc_internal.a /usr/local/lib/libabsl_time.a /usr/local/lib/libabsl_civil_time.a /usr/local/lib/libabsl_strings.a /usr/local/lib/libabsl_strings_internal.a -lrt /usr/local/lib/libabsl_base.a /usr/local/lib/libabsl_spinlock_wait.a /usr/local/lib/libabsl_int128.a /usr/local/lib/libabsl_throw_delegate.a /usr/local/lib/libabsl_time_zone.a /usr/local/lib/libabsl_bad_variant_access.a /usr/local/lib/libabsl_raw_logging_internal.a /usr/local/lib/libabsl_log_severity.a /usr/local/lib/libprotobuf.a /home/p4/Schreibtisch/t4p4s/P4Runtime_GRPCPP/static_lib/libp4rt.a -lstdc++ -Wl,--end-group

ld.lld-12: error: undefined symbol: nan
>>> referenced by charconv.cc
>>>               charconv.cc.o:(absl::lts_20211102::(anonymous namespace)::FloatTraits<double>::MakeNan(char const*)) in archive /usr/local/lib/libabsl_strings.a

ld.lld-12: error: undefined symbol: nanf
>>> referenced by charconv.cc
>>>               charconv.cc.o:(absl::lts_20211102::(anonymous namespace)::FloatTraits<float>::MakeNan(char const*)) in archive /usr/local/lib/libabsl_strings.a

ld.lld-12: error: undefined symbol: truncl
>>> referenced by int128.cc
>>>               int128.cc.o:(std::trunc(long double)) in archive /usr/local/lib/libabsl_int128.a

ld.lld-12: error: undefined symbol: ceil
>>> referenced by duration.cc
>>>               duration.cc.o:(double absl::lts_20211102::(anonymous namespace)::Round<double>(double)) in archive /usr/local/lib/libabsl_time.a
>>> referenced by ring_hash.cc
>>>               ring_hash.cc.o:(grpc_core::(anonymous namespace)::RingHash::Ring::Ring(grpc_core::(anonymous namespace)::RingHash*, grpc_core::RefCountedPtr<grpc_core::(anonymous namespace)::RingHash::RingHashSubchannelList>)) in archive /usr/local/lib/libgrpc.a
>>> referenced by ring_hash.cc
>>>               ring_hash.cc.o:(grpc_core::(anonymous namespace)::RingHash::Ring::Ring(grpc_core::(anonymous namespace)::RingHash*, grpc_core::RefCountedPtr<grpc_core::(anonymous namespace)::RingHash::RingHashSubchannelList>)) in archive /usr/local/lib/libgrpc.a

ld.lld-12: error: undefined symbol: floor
>>> referenced by duration.cc
>>>               duration.cc.o:(double absl::lts_20211102::(anonymous namespace)::Round<double>(double)) in archive /usr/local/lib/libabsl_time.a

ld.lld-12: error: undefined symbol: log2
>>> referenced by exponential_biased.cc
>>>               exponential_biased.cc.o:(absl::lts_20211102::profiling_internal::ExponentialBiased::GetSkipCount(long)) in archive /usr/local/lib/libabsl_exponential_biased.a
>>> referenced by flow_control.cc
>>>               flow_control.cc.o:(grpc_core::chttp2::TransportFlowControl::TargetLogBdp()) in archive /usr/local/lib/libgrpc.a

ld.lld-12: error: undefined symbol: log
>>> referenced by exponential_biased.cc
>>>               exponential_biased.cc.o:(absl::lts_20211102::profiling_internal::ExponentialBiased::GetSkipCount(long)) in archive /usr/local/lib/libabsl_exponential_biased.a

ld.lld-12: error: undefined symbol: rint
>>> referenced by exponential_biased.cc
>>>               exponential_biased.cc.o:(absl::lts_20211102::profiling_internal::ExponentialBiased::GetSkipCount(long)) in archive /usr/local/lib/libabsl_exponential_biased.a

ld.lld-12: error: undefined symbol: pow
>>> referenced by flow_control.cc
>>>               flow_control.cc.o:(grpc_core::chttp2::TransportFlowControl::PeriodicUpdate()) in archive /usr/local/lib/libgrpc.a

ld.lld-12: error: undefined symbol: logl
>>> referenced by rls.cc
>>>               rls.cc.o:(std::log(long double)) in archive /usr/local/lib/libgrpc.a

ld.lld-12: error: undefined symbol: nextafterf
>>> referenced by rls.cc
>>>               rls.cc.o:(std::nextafter(float, float)) in archive /usr/local/lib/libgrpc.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

(Terminating the parse helper process, port 51973, pid 8538)
Error: C compilation using ninja failed, see log build/l2switch@std-psa/log/22_ninja.txt (error code: 1)

It appears as though some kind of <math.h> or <cmath> cannot be found.
Here are the versions of the components. I'll add this info to Issue #41 as well.

- DPDK: 22.03.0
- GRPC: Commit from 2022-05-06 20:31:11 (SHA 5918f98ecbf5ace77f30fa97f7fc3e8bdac08e04)
- P4C: 1.2.2.1, Commit from 2022-03-01 18:37:15 (SHA 978a1166684010832a3bd408a7c68fd592ccb7a0). Local changes made to CMakeLists.txt by bootstrap script line 557 (changes `-fuse-ld=gold` to `-fuse-ld=lld-12`) 
- P4Runtime_GRPCPP: Commit from 2020-11-25 15:53:26 (SHA 261bdc73f149d15e61e73b4afa80c8085de9445f)
- PI: 0.1.0, Commit from 2022-03-09 22:04:59 (SHA f547455a260b710706bef82afab4cb9937bac416)
- Protobuf: 3.19.4, Commit from 2022-01-28 04:25:56 (SHA 22d0e265de7d2b3d2e9a00d071313502e7d4cccf)
- t4p4s: Commit from 2022-04-30 17:52:57 (SHA 22d0e265de7d2b3d2e9a00d071313502e7d4cccf). Local changes made to various files. None of them were made by me and I couldn't find a script that could have done them.

[msg-programs]

Update

The program compiles properly by adding the line marked with + to the meson.build.p4rt file found in the same directory as the t4p4s.sh. Here is the full modified file:

build_args += [
    '-L' + (grpcpp / 'static_lib'),
]

incdirs += [
    grpcpp,
    grpcpp / 'include',
    grpc / 'include',
    p4pi / 'proto/server',
]

all_dependencies += [
    dependency('grpc++'),
    dependency('grpc'),
    dependency('protobuf'),

    meson.get_compiler('c').find_library('p4rt', dirs: grpcpp / 'static_lib'),
    meson.get_compiler('c').find_library('stdc++'),
+   meson.get_compiler('c').find_library('m'),
]

I haven't created a pull request for this as I haven't had the time to properly test this, but it looks promising.

As it turns out, the t4p4s install that comes with the prebuild p4pi image does have this line in said file. Oddly enough, I couldn't find anything in the p4pi repo that would insert this line, be it a script or a commit. This line never existed in the t4p4s repo as far as I can tell either, so it probably was added manually before building the image.