[META] Replace Bouncycastle dependencies with FIPS counterparts

[beanuwave]

Please describe the end goal of this project

Replacing all Bouncy Castle libraries with their FIPS equivalents reduces complexity and eliminates the additional build step whenever switching between FIPS / non-FIPS modes. All functionality currently provided by the BC libs can be fully covered by combination of SUN, Bouncy Castle FIPS (running in general mode), Password4j, rfksystems' Blake2b.

Following repositories register BC libs directly and require maintenance:

https://github.com/opensearch-project/OpenSearch
https://github.com/opensearch-project/security
https://github.com/opensearch-project/performance-analyzer-rca
https://github.com/opensearch-project/performance-analyzer
https://github.com/opensearch-project/flow-framework
https://github.com/opensearch-project/opensearch-oci-object-storage
https://github.com/opensearch-project/sql
https://github.com/opensearch-project/ml-commons
https://github.com/opensearch-project/opensearch-metrics

Tasks

Overall:

• replace BC with BCFIPS libs and rely on libs.versions.toml file for correct version number.
• discover behavioral differences.

OpenSearch core:

• replace BC as security provider with combination of SUN + BCFIPS providers.
• rely on password4j lib for bcrypt password hashing.

Security plugin:

• switch the security provider BC -> BCFIPS.
• Remove NamedCurve from shibboleth's OpenSAML implementation.

SQL plugin:

• ensure com.amazonaws:aws-encryption-sdk-java can run with BCFIPS lib, or can be replaced with something else.

Alternative

Just to mention the alternative approach "Switch between BC/BCFIPS at build time" :

PROS:

• No changes on components from above are required, as long as they are actively excluded to run in FIPS mode. Because both Bouncy Castle libraries cannot be on the classpath at the same time.
• No expected performance implications.

CONS:

• Requires extra effort to build the application with swapped libraries.
• Must address behavioral differences, such as error handling or checksum matching.
• Necessitates dual code paths wherever the provider does not match the same classpath or specific features of the BC provider.
• Increases the potential risk of misconfiguration, which might lead to building against the wrong provider.

Supporting References

RFC: #4254
RFC #3420

Issues

• Migrate from BC to BCFIPS libraries #17507
• [BUG] Current version of OpenSAML is incompatible with the Bouncy Castle FIPS library

Related component

Other

[cwperks]

Thank you for filing this issue @beanuwave. Out of the components listed, SQL, ML-Commons, and Flow Framework and Security need to be shown that existing functionality works with BCFIPS jars since these are plugins in the default distribution of OpenSearch.

I believe PA is set to be removed, but if its still included in upcoming versions then that would need to be shown to work with BCFIPS jars.

One other question I have is that @terryquigleysas mentioned that BCFIPS and non-FIPS jars cannot be on the same classpath, but is it possible for core to have FIPS dependencies while plugins (at least in the interim) have non-FIPS dependencies? My understanding is that plugins are loaded with a separate classLoader.