Skip to content

Commit b9ae877

Browse files
beanuwaveiigoninbennygoerzigKarstenSchnitterKai Sternad
authored and
anandpatel9998
committed
Add build-tooling to run in the FIPS environment (opensearch-project#18921)
* Add build-tooling to run in the FIPS environment Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * replace java.util with lfs4j logger; use AccessController to read cacerts file; add bc-jsse provider Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * replace fallback mechanism with cluster-setting to determine the strategy for default trust-store Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * cleanup changelog.md Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * add default enum value; extend MultiProviderTrustStoreHandler test-class Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * fix :server:forbiddenApisTest Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * configure docker and testClusters builds to use generated truststore Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * add more unit tests Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * replace MultiProviderTrustStoreHandler with demo script Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * fix CHANGELOG; update CLI's build.gradle; write additional docu Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * add more tests Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * remove '--enable-native-access=ALL-UNNAMED' Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * apply BouncyCastleThreadFilter Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * replace BuildParams.inFipsJvm with getter Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * make use of 'testFipsRuntimeOnly' inside StandaloneRestTestPlugin; override and simplify ConfigurationProperties#toString Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * use BCFIPS_RNG; make java.util.Scanner test-friendly; write additional unit tests Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * get SecureRandom from server-module Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * add '--password option' to CLI Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * increase tests coverage for FipsTrustStoreValidator Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * fix teardown on SecurityProviderManagerTests Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * Throw a reasonable error when FIPS is not configured Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * Refactor tests to use TemporaryFolder rule; simplify and clean up input/output handling in CLI classes Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> * Rename `SecurityProviderService` to `SecurityConfigurationPrinter` for improved clarity Signed-off-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]> --------- Signed-off-by: Igonin <[email protected]> Co-authored-by: Igonin <[email protected]> Co-authored-by: Benny Goerzig <[email protected]> Co-authored-by: Karsten Schnitter <[email protected]> Co-authored-by: Kai Sternad <[email protected]>
1 parent 37e5703 commit b9ae877

File tree

79 files changed

+4018
-69
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

79 files changed

+4018
-69
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1818
- Introduced internal API for retrieving metadata about requested indices from transport actions ([#18523](https://github.com/opensearch-project/OpenSearch/pull/18523))
1919
- Add Hybrid Cardinality collector to prioritize Ordinals Collector ([#19524](https://github.com/opensearch-project/OpenSearch/pull/19524))
2020
- Add cluster defaults for merge autoThrottle, maxMergeThreads, and maxMergeCount; Add segment size filter to the merged segment warmer ([#19629](https://github.com/opensearch-project/OpenSearch/pull/19629))
21+
- Add build-tooling to run in FIPS environment ([#18921](https://github.com/opensearch-project/OpenSearch/pull/18921))
2122
- Add SMILE/CBOR/YAML document format support to Bulk GRPC endpoint ([#19744](https://github.com/opensearch-project/OpenSearch/pull/19744))
2223

2324
### Changed

‎build.gradle‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,11 @@ apply from: 'gradle/run.gradle'
7171
apply from: 'gradle/missing-javadoc.gradle'
7272
apply from: 'gradle/code-coverage.gradle'
7373

74+
// Apply FIPS configuration to all projects
75+
allprojects {
76+
apply from: "$rootDir/gradle/fips.gradle"
77+
}
78+
7479
// common maven publishing configuration
7580
allprojects {
7681
group = 'org.opensearch'
@@ -421,8 +426,12 @@ gradle.projectsEvaluated {
421426
dependsOn(project(':libs:agent-sm:agent').prepareAgent)
422427
jvmArgs += ["-javaagent:" + project(':libs:agent-sm:agent').jar.archiveFile.get()]
423428
}
424-
if (BuildParams.inFipsJvm) {
425-
task.jvmArgs += ["-Dorg.bouncycastle.fips.approved_only=true"]
429+
if (BuildParams.isInFipsJvm()) {
430+
def fipsSecurityFile = project.rootProject.file('distribution/src/config/fips_java.security')
431+
task.jvmArgs += [
432+
"-Dorg.bouncycastle.fips.approved_only=true",
433+
"-Djava.security.properties=${fipsSecurityFile}"
434+
]
426435
}
427436
}
428437
}
@@ -693,14 +702,6 @@ allprojects {
693702
plugins.withId('lifecycle-base') {
694703
checkPart1.configure { dependsOn 'check' }
695704
}
696-
697-
plugins.withId('opensearch.testclusters') {
698-
testClusters.configureEach {
699-
if (BuildParams.inFipsJvm) {
700-
keystorePassword 'notarealpasswordphrase'
701-
}
702-
}
703-
}
704705
}
705706

706707
subprojects {

‎buildSrc/build.gradle‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ if (project != rootProject) {
193193
disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest', 'forbiddenApisTestFixtures')
194194
jarHell.enabled = false
195195
thirdPartyAudit.enabled = false
196-
if (org.opensearch.gradle.info.BuildParams.inFipsJvm) {
196+
if (org.opensearch.gradle.info.BuildParams.isInFipsJvm()) {
197197
// We don't support running gradle with a JVM that is in FIPS 140 mode, so we don't test it.
198198
// WaitForHttpResourceTests tests would fail as they use JKS/PKCS12 keystores
199199
test.enabled = false
@@ -264,7 +264,7 @@ if (project != rootProject) {
264264
useJUnitPlatform()
265265
inputs.dir(file("src/testKit")).withPropertyName("testkit dir").withPathSensitivity(PathSensitivity.RELATIVE)
266266
systemProperty 'test.version_under_test', version
267-
onlyIf { org.opensearch.gradle.info.BuildParams.inFipsJvm == false }
267+
onlyIf { org.opensearch.gradle.info.BuildParams.isInFipsJvm() == false }
268268
maxParallelForks = System.getProperty('tests.jvms', org.opensearch.gradle.info.BuildParams.defaultParallel.toString()) as Integer
269269
testClassesDirs = sourceSets.integTest.output.classesDirs
270270
classpath = sourceSets.integTest.runtimeClasspath

‎buildSrc/src/main/groovy/org/opensearch/gradle/test/ClusterFormationTasks.groovy‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -761,7 +761,7 @@ class ClusterFormationTasks {
761761
start.doLast(opensearchRunner)
762762
start.doFirst {
763763
// If the node runs in a FIPS 140-2 JVM, the BCFKS default keystore will be password protected
764-
if (BuildParams.inFipsJvm) {
764+
if (BuildParams.isInFipsJvm()) {
765765
node.config.systemProperties.put('javax.net.ssl.trustStorePassword', 'password')
766766
node.config.systemProperties.put('javax.net.ssl.keyStorePassword', 'password')
767767
}

‎buildSrc/src/main/groovy/org/opensearch/gradle/test/StandaloneRestTestPlugin.groovy‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@
3131
package org.opensearch.gradle.test
3232

3333
import groovy.transform.CompileStatic
34+
import org.gradle.api.artifacts.VersionCatalog
35+
import org.gradle.api.artifacts.VersionCatalogsExtension
3436
import org.opensearch.gradle.OpenSearchJavaPlugin
3537
import org.opensearch.gradle.ExportOpenSearchBuildResourcesTask
3638
import org.opensearch.gradle.RepositoriesSetupPlugin
@@ -92,6 +94,10 @@ class StandaloneRestTestPlugin implements Plugin<Project> {
9294
// create a compileOnly configuration as others might expect it
9395
project.configurations.create("compileOnly")
9496
project.dependencies.add('testImplementation', project.project(':test:framework'))
97+
if (BuildParams.isInFipsJvm()) {
98+
VersionCatalog libs = project.extensions.getByType(VersionCatalogsExtension).named("libs")
99+
project.dependencies.add('testFipsRuntimeOnly', libs.findBundle("bouncycastle").get())
100+
}
95101

96102
EclipseModel eclipse = project.extensions.getByType(EclipseModel)
97103
eclipse.classpath.sourceSets = [testSourceSet]

‎buildSrc/src/main/java/org/opensearch/gradle/test/rest/RestTestUtil.java‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,18 @@ static void setupDependencies(Project project, SourceSet sourceSet) {
102102
);
103103
}
104104

105+
if (BuildParams.isInFipsJvm()) {
106+
project.getDependencies()
107+
.add(
108+
sourceSet.getImplementationConfigurationName(),
109+
"org.bouncycastle:bc-fips:" + VersionProperties.getVersions().get("bouncycastle_jce")
110+
);
111+
project.getDependencies()
112+
.add(
113+
sourceSet.getImplementationConfigurationName(),
114+
"org.bouncycastle:bctls-fips:" + VersionProperties.getVersions().get("bouncycastle_tls")
115+
);
116+
}
105117
}
106118

107119
}

‎buildSrc/src/main/resources/opensearch-fips-truststore.bcfks‎

182 KB
Binary file not shown.

‎client/test/src/main/java/org/opensearch/client/BouncyCastleThreadFilter.java‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.client;
10+
11+
import com.carrotsearch.randomizedtesting.ThreadFilter;
12+
13+
/**
14+
* ThreadFilter to exclude ThreadLeak checks for BC’s global background threads
15+
*
16+
* <p>clone from the original, which is located in ':test:framework'</p>
17+
*/
18+
public class BouncyCastleThreadFilter implements ThreadFilter {
19+
@Override
20+
public boolean reject(Thread t) {
21+
String n = t.getName();
22+
// Ignore BC’s global background threads
23+
return "BC Disposal Daemon".equals(n) || "BC Cleanup Executor".equals(n);
24+
}
25+
}

‎client/test/src/main/java/org/opensearch/client/RestClientTestCase.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
import com.carrotsearch.randomizedtesting.annotations.SeedDecorators;
3939
import com.carrotsearch.randomizedtesting.annotations.TestMethodProviders;
4040
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakAction;
41+
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
4142
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakGroup;
4243
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakLingering;
4344
import com.carrotsearch.randomizedtesting.annotations.ThreadLeakScope;
@@ -65,6 +66,7 @@
6566
@ThreadLeakAction({ ThreadLeakAction.Action.WARN, ThreadLeakAction.Action.INTERRUPT })
6667
@ThreadLeakZombies(ThreadLeakZombies.Consequence.IGNORE_REMAINING_TESTS)
6768
@ThreadLeakLingering(linger = 5000) // 5 sec lingering
69+
@ThreadLeakFilters(filters = BouncyCastleThreadFilter.class)
6870
@TimeoutSuite(millis = 2 * 60 * 60 * 1000)
6971
public abstract class RestClientTestCase extends RandomizedTest {
7072

‎distribution/build.gradle‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -312,7 +312,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
312312
* Properties to expand when copying packaging files *
313313
*****************************************************************************/
314314
configurations {
315-
['libs', 'libsPluginCli', 'libsKeystoreCli', 'bcFips'].each {
315+
['libs', 'libsPluginCli', 'libsKeystoreCli', 'libsFipsInstallerCli', 'bcFips'].each {
316316
create(it) {
317317
canBeConsumed = false
318318
canBeResolved = true
@@ -333,6 +333,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
333333

334334
libsPluginCli project(':distribution:tools:plugin-cli')
335335
libsKeystoreCli project(path: ':distribution:tools:keystore-cli')
336+
libsFipsInstallerCli project(path: ':distribution:tools:fips-demo-installer-cli')
336337

337338
bcFips libs.bundles.bouncycastle
338339
}
@@ -346,7 +347,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
346347
copySpec {
347348
// delay by using closures, since they have not yet been configured, so no jar task exists yet
348349
from(configurations.libs)
349-
if ( BuildParams.inFipsJvm ) {
350+
if ( BuildParams.isInFipsJvm() ) {
350351
from(configurations.bcFips)
351352
}
352353
into('tools/plugin-cli') {
@@ -355,6 +356,10 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
355356
into('tools/keystore-cli') {
356357
from(configurations.libsKeystoreCli)
357358
}
359+
// Add FIPS installer CLI
360+
into('tools/fips-demo-installer-cli') {
361+
from(configurations.libsFipsInstallerCli)
362+
}
358363
}
359364
}
360365

0 commit comments

Comments
 (0)