[BUG] libsrt.so leaks internal and statically-linked OpenSSL symbols into global dynamic linker namespace

[eugenevt]

When libsrt.so compiled wit static OpenSSL is loaded alongside system OpenSSL (e.g. OpenSSL 3.x on Ubuntu 22.04), the dynamic linker encounters symbol collisions between OpenSSL symbols statically baked into libsrt.so and the system-provided libssl.so/libcrypto.so. This causes unpredictable crashes depending on which library's symbol the linker resolves first at runtime.

1. Build libsrt.so with static OpenSSL.
2. Use nm -D --defines-only libsrt.so to get exported symbols.
3. Observe symbols like AES_* and others.

Expected that only symbols marked as SRT_API is visible.

CMake build script should contain "-fvisibilty=hidden" for compiler flags and "-Wl,--exclude-libs,libcrypto.a:libssl.a" for the case of static OpenSSL.

• OS: Ubuntu 20.04/22.04
• SRT 1.5.4

[ethouris]

This is where the static openssl case is recognized in CMakeLists.txt file:

			if (SRT_USE_OPENSSL_STATIC_LIBS)
				# use `pkg-config --static xxx` found libs
				set(OPENSSL_USE_STATIC_LIBS True)
				set(OPENSSL_MSVC_STATIC_RT True)
			endif()

This part sets it on demand, but OPENSSL_USE_STATIC_LIBS can be likely also changed by the cmake itself upon detection.

Likely you'd have to check later if OPENSSL_USE_STATIC_LIBS is true, as well as if HAVE_COMPILER_GNU_COMPAT, then add these flags through target_compile_options(srt_virtual .... ) - this has to be done after srt_virtual was defined through add_library call.

Please check if this solution would work correctly, and if so, you can submit a PR - or, if this is too much trouble, paste the patch file here, I'll do the rest.