Skip to content

Commit 32d3e97

Browse files
amitgalitzamitgalitz
committed
adding ability to run AD with 2 local clusters
Signed-off-by: Amit Galitzky <[email protected]>
1 parent 41db8c0 commit 32d3e97

6 files changed

+132
-65
lines changed

‎.github/workflows/changelog_verifier.yml

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ on:
44
branches-ignore:
55
- 'whitesource-remediate/**'
66
- 'backport/**'
7+
- 'dependabot/**'
78
pull_request:
89
types: [opened, edited, review_requested, synchronize, reopened, ready_for_review, labeled, unlabeled]
910

‎.github/workflows/labeler.yml

-3
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,6 @@ on:
66
types:
77
- opened
88

9-
env:
10-
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
11-
129
jobs:
1310
label:
1411
runs-on: ubuntu-latest

‎.github/workflows/test_security.yml

+1-4
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,6 @@ on:
77
branches:
88
- "*"
99

10-
env:
11-
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
12-
1310
jobs:
1411
Build-ad:
1512
strategy:
@@ -22,7 +19,7 @@ jobs:
2219

2320
steps:
2421
- name: Setup Java ${{ matrix.java }}
25-
uses: actions/setup-java@v3
22+
uses: actions/setup-java@v4
2623
with:
2724
distribution: 'temurin'
2825
java-version: ${{ matrix.java }}

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
88
### Enhancements
99
### Bug Fixes
1010
### Infrastructure
11+
- Adding dual cluster arg to gradle run ([#1441](https://github.com/opensearch-project/anomaly-detection/pull/1441))
1112
### Documentation
1213
### Maintenance
1314
### Refactoring

‎DEVELOPER_GUIDE.md

+11-10
Original file line numberDiff line numberDiff line change
@@ -41,16 +41,17 @@ Currently we just put RCF jar in lib as dependency. Plan to publish to Maven and
4141

4242
1. `./gradlew build` builds and tests
4343
2. `./gradlew :run` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed
44-
3. `./gradlew :integTest` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed and runs all integration tests except security
45-
4. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
46-
5. `./gradlew integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=<admin-password>` launches integration tests against a local cluster and run tests with security
47-
6. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
48-
7. `./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler creating a mixed cluster.
49-
8. `./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with anomaly-detection and job-scheduler.
50-
9. `./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler.
51-
10. `./gradlew bwcTestSuite -Dtests.security.manager=false` runs all the above bwc tests combined.
52-
11. `./gradlew ':test' --tests "org.opensearch.ad.ml.HCADModelPerfTests" -Dtests.seed=2AEBDBBAE75AC5E0 -Dtests.security.manager=false -Dtests.locale=es-CU -Dtests.timezone=Chile/EasterIsland -Dtest.logs=true -Dmodel-benchmark=true` launches HCAD model performance tests and logs the result in the standard output
53-
12. `./gradlew integTest --tests "org.opensearch.ad.e2e.SingleStreamModelPerfIT" -Dtests.seed=60CDDB34427ACD0C -Dtests.security.manager=false -Dtests.locale=kab-DZ -Dtests.timezone=Asia/Hebron -Dtest.logs=true -Dmodel-benchmark=true` launches single stream AD model performance tests and logs the result in the standard output
44+
3. `./gradlew :run -PdualCluster=true` launches 2 single node clusters with anomaly-detection (and job-scheduler) plugin installed, one cluster is on localhost:9200 and the other on localhost:9200
45+
4. `./gradlew :integTest` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed and runs all integration tests except security
46+
5. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
47+
6. `./gradlew integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=<admin-password>` launches integration tests against a local cluster and run tests with security
48+
7. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
49+
8. `./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler creating a mixed cluster.
50+
9. `./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with anomaly-detection and job-scheduler.
51+
10. `./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler.
52+
11. `./gradlew bwcTestSuite -Dtests.security.manager=false` runs all the above bwc tests combined.
53+
12. `./gradlew ':test' --tests "org.opensearch.ad.ml.HCADModelPerfTests" -Dtests.seed=2AEBDBBAE75AC5E0 -Dtests.security.manager=false -Dtests.locale=es-CU -Dtests.timezone=Chile/EasterIsland -Dtest.logs=true -Dmodel-benchmark=true` launches HCAD model performance tests and logs the result in the standard output
54+
13. `./gradlew integTest --tests "org.opensearch.ad.e2e.SingleStreamModelPerfIT" -Dtests.seed=60CDDB34427ACD0C -Dtests.security.manager=false -Dtests.locale=kab-DZ -Dtests.timezone=Asia/Hebron -Dtest.logs=true -Dmodel-benchmark=true` launches single stream AD model performance tests and logs the result in the standard output
5455

5556
When launching a cluster using one of the above commands logs are placed in `/build/cluster/run node0/opensearch-<version>/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.
5657

‎build.gradle

+118-48
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ import java.util.concurrent.Callable
1313
import org.opensearch.gradle.test.RestIntegTestTask
1414
import org.opensearch.gradle.testclusters.StandaloneRestIntegTestTask
1515
import org.opensearch.gradle.info.BuildParams
16+
import java.util.stream.Collectors
17+
1618

1719
buildscript {
1820
ext {
@@ -48,7 +50,7 @@ buildscript {
4850
jacksonVersion = "2.18.2"
4951
// gradle build won't print logs during test by default unless there is a failure.
5052
// It is useful to record intermediately information like prediction precision and recall.
51-
// This option turn on log printing during tests.
53+
// This option turn on log printing during tests.
5254
printLogs = "true" == System.getProperty("test.logs", "false")
5355
}
5456

@@ -227,7 +229,7 @@ configurations.all {
227229

228230
force "com.google.guava:guava:32.1.3-jre" // CVE for 31.1
229231
force("com.fasterxml.jackson.core:jackson-core:${jacksonVersion}")
230-
force "org.eclipse.platform:org.eclipse.core.runtime:3.29.0" // CVE for < 3.29.0
232+
force "org.eclipse.platform:org.eclipse.core.runtime:3.29.0" // CVE for < 3.29.0
231233
}
232234
}
233235

@@ -254,7 +256,7 @@ publishing {
254256
}
255257
}
256258
}
257-
259+
258260
repositories {
259261
maven {
260262
name = "Snapshots"
@@ -397,10 +399,30 @@ integTest {
397399
}
398400
}
399401

400-
testClusters.integTest {
401-
testDistribution = "ARCHIVE"
402-
// Cluster shrink exception thrown if we try to set numberOfNodes to 1, so only apply if > 1
403-
if (_numNodes > 1) numberOfNodes = _numNodes
402+
403+
404+
testClusters {
405+
leaderCluster {
406+
testDistribution = "ARCHIVE"
407+
setting 'path.repo', file("${buildDir}/leaderCluster/repo").absolutePath
408+
setting 'discovery.type', 'single-node'
409+
// Add custom attribute to identify this cluster
410+
setting 'node.attr.cluster_role', 'leader'
411+
configureClusterPlugins(delegate, _numNodes)
412+
}
413+
414+
followCluster {
415+
testDistribution = "ARCHIVE"
416+
setting 'path.repo', file("${buildDir}/followCluster/repo").absolutePath
417+
setting 'discovery.type', 'single-node'
418+
// Add custom attribute to identify this cluster
419+
setting 'node.attr.cluster_role', 'follower'
420+
configureClusterPlugins(delegate, _numNodes)
421+
}
422+
}
423+
424+
def configureClusterPlugins(cluster, _numNodes) {
425+
404426
// When running integration tests it doesn't forward the --debug-jvm to the cluster anymore
405427
// i.e. we have to use a custom property to flag when we want to debug elasticsearch JVM
406428
// since we also support multi node integration tests we increase debugPort per node
@@ -411,42 +433,52 @@ testClusters.integTest {
411433
debugPort += 1
412434
}
413435
}
414-
plugin(project.tasks.bundlePlugin.archiveFile)
415-
plugin(provider(new Callable<RegularFile>(){
416-
@Override
417-
RegularFile call() throws Exception {
418-
return new RegularFile() {
419-
@Override
420-
File getAsFile() {
421-
return configurations.zipArchive.asFileTree.getSingleFile()
436+
cluster.with {
437+
plugin(project.tasks.bundlePlugin.archiveFile)
438+
plugin(provider(new Callable<RegularFile>(){
439+
@Override
440+
RegularFile call() throws Exception {
441+
return new RegularFile() {
442+
@Override
443+
File getAsFile() {
444+
return configurations.zipArchive.asFileTree.getSingleFile()
445+
}
422446
}
423447
}
448+
}))
449+
450+
// As of ES 7.7.0 the opendistro-anomaly-detection plugin is being added to the list of plugins for the testCluster during build before
451+
// the opensearch-job-scheduler plugin, which is causing build failures. From the stack trace, this looks like a bug.
452+
//
453+
// Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [opensearch-job-scheduler], dependency of [opendistro-anomaly-detection]
454+
// at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:452)
455+
//
456+
// One explanation is that ES build script sort plugins according to the natural ordering of their names.
457+
// opendistro-anomaly-detection comes before opensearch-job-scheduler.
458+
//
459+
// The following is a comparison of different plugin installation order:
460+
// Before 7.7:
461+
// ./bin/elasticsearch-plugin install --batch file:opendistro-anomaly-detection.zip file:opensearch-job-scheduler.zip
462+
//
463+
// After 7.7:
464+
// ./bin/elasticsearch-plugin install --batch file:opensearch-job-scheduler.zip file:opendistro-anomaly-detection.zip
465+
//
466+
// A temporary hack is to reorder the plugins list after evaluation but prior to task execution when the plugins are installed.
467+
nodes.each { node ->
468+
def plugins = node.plugins
469+
def firstPlugin = plugins.get(0)
470+
plugins.remove(0)
471+
plugins.add(firstPlugin)
424472
}
425-
}))
426-
427-
// As of ES 7.7.0 the opendistro-anomaly-detection plugin is being added to the list of plugins for the testCluster during build before
428-
// the opensearch-job-scheduler plugin, which is causing build failures. From the stack trace, this looks like a bug.
429-
//
430-
// Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [opensearch-job-scheduler], dependency of [opendistro-anomaly-detection]
431-
// at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:452)
432-
//
433-
// One explanation is that ES build script sort plugins according to the natural ordering of their names.
434-
// opendistro-anomaly-detection comes before opensearch-job-scheduler.
435-
//
436-
// The following is a comparison of different plugin installation order:
437-
// Before 7.7:
438-
// ./bin/elasticsearch-plugin install --batch file:opendistro-anomaly-detection.zip file:opensearch-job-scheduler.zip
439-
//
440-
// After 7.7:
441-
// ./bin/elasticsearch-plugin install --batch file:opensearch-job-scheduler.zip file:opendistro-anomaly-detection.zip
442-
//
443-
// A temporary hack is to reorder the plugins list after evaluation but prior to task execution when the plugins are installed.
444-
nodes.each { node ->
445-
def plugins = node.plugins
446-
def firstPlugin = plugins.get(0)
447-
plugins.remove(0)
448-
plugins.add(firstPlugin)
449473
}
474+
475+
}
476+
477+
testClusters.integTest {
478+
testDistribution = "ARCHIVE"
479+
// Cluster shrink exception thrown if we try to set numberOfNodes to 1, so only apply if > 1
480+
if (_numNodes > 1) numberOfNodes = _numNodes
481+
configureClusterPlugins(delegate, this.hasProperty('numberOfNodes') ? numberOfNodes : _numNodes)
450482
}
451483

452484
task integTestRemote(type: RestIntegTestTask) {
@@ -475,7 +507,7 @@ task integTestRemote(type: RestIntegTestTask) {
475507
}
476508
}
477509

478-
2.times {i ->
510+
2.times {i ->
479511
testClusters {
480512
"${baseName}$i" {
481513
testDistribution = "ARCHIVE"
@@ -548,8 +580,8 @@ List<Provider<RegularFile>> plugins = [
548580
})
549581
]
550582

551-
// Creates 2 test clusters with 3 nodes of the old version.
552-
2.times {i ->
583+
// Creates 2 test clusters with 3 nodes of the old version.
584+
2.times {i ->
553585
task "${baseName}#oldVersionClusterTask$i"(type: RestIntegTestTask) {
554586
useCluster testClusters."${baseName}$i"
555587
filter {
@@ -560,10 +592,10 @@ List<Provider<RegularFile>> plugins = [
560592
systemProperty 'tests.plugin_bwc_version', bwcVersion
561593
nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}$i".allHttpSocketURI.join(",")}")
562594
nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}$i".getName()}")
563-
}
595+
}
564596
}
565597

566-
// Upgrades one node of the old cluster to new OpenSearch version with upgraded plugin version
598+
// Upgrades one node of the old cluster to new OpenSearch version with upgraded plugin version
567599
// This results in a mixed cluster with 2 nodes on the old version and 1 upgraded node.
568600
// This is also used as a one third upgraded cluster for a rolling upgrade.
569601
task "${baseName}#mixedClusterTask"(type: RestIntegTestTask) {
@@ -621,7 +653,7 @@ task "${baseName}#rollingUpgradeClusterTask"(type: RestIntegTestTask) {
621653
nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}0".getName()}")
622654
}
623655

624-
// Upgrades all the nodes of the old cluster to new OpenSearch version with upgraded plugin version
656+
// Upgrades all the nodes of the old cluster to new OpenSearch version with upgraded plugin version
625657
// at the same time resulting in a fully upgraded cluster.
626658
task "${baseName}#fullRestartClusterTask"(type: RestIntegTestTask) {
627659
dependsOn "${baseName}#oldVersionClusterTask1"
@@ -649,16 +681,54 @@ task bwcTestSuite(type: RestIntegTestTask) {
649681

650682
run {
651683
doFirst {
652-
// There seems to be an issue when running multi node run or integ tasks with unicast_hosts
653-
// not being written, the waitForAllConditions ensures it's written
654684
getClusters().forEach { cluster ->
655685
cluster.waitForAllConditions()
656686
}
687+
688+
if (project.hasProperty('dualCluster')) {
689+
def clusterPorts = [:] // Map to store leader/follow cluster info
690+
691+
getClusters().forEach { cluster ->
692+
def clusterName = cluster.getName()
693+
def allTransportSocketURI = cluster.nodes
694+
.collectMany { node -> node.getAllTransportPortURI() }
695+
.join(",")
696+
697+
def allHttpSocketURI = cluster.nodes
698+
.collectMany { node -> node.getAllHttpSocketURI() }
699+
.join(",")
700+
701+
clusterPorts[clusterName] = [
702+
httpHosts: allHttpSocketURI,
703+
transportHosts: allTransportSocketURI
704+
]
705+
706+
println "${clusterName.capitalize()} cluster running at HTTP: ${allHttpSocketURI}, Transport: ${allTransportSocketURI}"
707+
}
708+
709+
// Pass cluster info to tests
710+
if (clusterPorts.containsKey('leader') && clusterPorts.containsKey('follow')) {
711+
systemProperty "tests.leader.cluster.name", "leader"
712+
systemProperty "tests.follow.cluster.name", "follower"
713+
714+
systemProperty "tests.cluster.leader.http_hosts", clusterPorts['leader'].httpHosts
715+
systemProperty "tests.cluster.follow.http_hosts", clusterPorts['follow'].httpHosts
716+
717+
systemProperty "tests.cluster.leader.transport_hosts", clusterPorts['leader'].transportHosts
718+
systemProperty "tests.cluster.follow.transport_hosts", clusterPorts['follow'].transportHosts
719+
}
720+
}
657721
}
658722

659-
useCluster testClusters.integTest
723+
if (project.hasProperty('dualCluster')) {
724+
useCluster testClusters.leaderCluster
725+
useCluster testClusters.followCluster
726+
} else {
727+
useCluster testClusters.integTest
728+
}
660729
}
661730

731+
662732
evaluationDependsOnChildren()
663733

664734
task release(type: Copy, group: 'build') {

0 commit comments

Comments
 (0)