Skip to content

Commit da650a6

Browse files
amitgalitzamitgalitz
committed
adding ability to run AD with 2 local clusters
Signed-off-by: Amit Galitzky <[email protected]>
1 parent 41db8c0 commit da650a6

6 files changed

+129
-63
lines changed

‎.github/workflows/changelog_verifier.yml

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ on:
44
branches-ignore:
55
- 'whitesource-remediate/**'
66
- 'backport/**'
7+
- 'dependabot/**'
78
pull_request:
89
types: [opened, edited, review_requested, synchronize, reopened, ready_for_review, labeled, unlabeled]
910

‎.github/workflows/labeler.yml

-3
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,6 @@ on:
66
types:
77
- opened
88

9-
env:
10-
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
11-
129
jobs:
1310
label:
1411
runs-on: ubuntu-latest

‎.github/workflows/test_security.yml

+1-4
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,6 @@ on:
77
branches:
88
- "*"
99

10-
env:
11-
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
12-
1310
jobs:
1411
Build-ad:
1512
strategy:
@@ -22,7 +19,7 @@ jobs:
2219

2320
steps:
2421
- name: Setup Java ${{ matrix.java }}
25-
uses: actions/setup-java@v3
22+
uses: actions/setup-java@v4
2623
with:
2724
distribution: 'temurin'
2825
java-version: ${{ matrix.java }}

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
88
### Enhancements
99
### Bug Fixes
1010
### Infrastructure
11+
- Adding dual cluster arg to gradle run ([#1441](https://github.com/opensearch-project/anomaly-detection/pull/1441))
1112
### Documentation
1213
### Maintenance
1314
### Refactoring

‎DEVELOPER_GUIDE.md

+11-10
Original file line numberDiff line numberDiff line change
@@ -41,16 +41,17 @@ Currently we just put RCF jar in lib as dependency. Plan to publish to Maven and
4141

4242
1. `./gradlew build` builds and tests
4343
2. `./gradlew :run` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed
44-
3. `./gradlew :integTest` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed and runs all integration tests except security
45-
4. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
46-
5. `./gradlew integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=<admin-password>` launches integration tests against a local cluster and run tests with security
47-
6. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
48-
7. `./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler creating a mixed cluster.
49-
8. `./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with anomaly-detection and job-scheduler.
50-
9. `./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler.
51-
10. `./gradlew bwcTestSuite -Dtests.security.manager=false` runs all the above bwc tests combined.
52-
11. `./gradlew ':test' --tests "org.opensearch.ad.ml.HCADModelPerfTests" -Dtests.seed=2AEBDBBAE75AC5E0 -Dtests.security.manager=false -Dtests.locale=es-CU -Dtests.timezone=Chile/EasterIsland -Dtest.logs=true -Dmodel-benchmark=true` launches HCAD model performance tests and logs the result in the standard output
53-
12. `./gradlew integTest --tests "org.opensearch.ad.e2e.SingleStreamModelPerfIT" -Dtests.seed=60CDDB34427ACD0C -Dtests.security.manager=false -Dtests.locale=kab-DZ -Dtests.timezone=Asia/Hebron -Dtest.logs=true -Dmodel-benchmark=true` launches single stream AD model performance tests and logs the result in the standard output
44+
3. `./gradlew :run -PdualCluster=true` launches 2 single node clusters with anomaly-detection (and job-scheduler) plugin installed, one cluster is on localhost:9200 and the other on localhost:9200
45+
4. `./gradlew :integTest` launches a single node cluster with anomaly-detection (and job-scheduler) plugin installed and runs all integration tests except security
46+
5. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
47+
6. `./gradlew integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=<admin-password>` launches integration tests against a local cluster and run tests with security
48+
7. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
49+
8. `./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler creating a mixed cluster.
50+
9. `./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with anomaly-detection and job-scheduler.
51+
10. `./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler.
52+
11. `./gradlew bwcTestSuite -Dtests.security.manager=false` runs all the above bwc tests combined.
53+
12. `./gradlew ':test' --tests "org.opensearch.ad.ml.HCADModelPerfTests" -Dtests.seed=2AEBDBBAE75AC5E0 -Dtests.security.manager=false -Dtests.locale=es-CU -Dtests.timezone=Chile/EasterIsland -Dtest.logs=true -Dmodel-benchmark=true` launches HCAD model performance tests and logs the result in the standard output
54+
13. `./gradlew integTest --tests "org.opensearch.ad.e2e.SingleStreamModelPerfIT" -Dtests.seed=60CDDB34427ACD0C -Dtests.security.manager=false -Dtests.locale=kab-DZ -Dtests.timezone=Asia/Hebron -Dtest.logs=true -Dmodel-benchmark=true` launches single stream AD model performance tests and logs the result in the standard output
5455

5556
When launching a cluster using one of the above commands logs are placed in `/build/cluster/run node0/opensearch-<version>/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.
5657

‎build.gradle

+115-46
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,8 @@ import java.util.concurrent.Callable
1313
import org.opensearch.gradle.test.RestIntegTestTask
1414
import org.opensearch.gradle.testclusters.StandaloneRestIntegTestTask
1515
import org.opensearch.gradle.info.BuildParams
16+
import java.util.stream.Collectors
17+
1618

1719
buildscript {
1820
ext {
@@ -48,7 +50,7 @@ buildscript {
4850
jacksonVersion = "2.18.2"
4951
// gradle build won't print logs during test by default unless there is a failure.
5052
// It is useful to record intermediately information like prediction precision and recall.
51-
// This option turn on log printing during tests.
53+
// This option turn on log printing during tests.
5254
printLogs = "true" == System.getProperty("test.logs", "false")
5355
}
5456

@@ -227,7 +229,7 @@ configurations.all {
227229

228230
force "com.google.guava:guava:32.1.3-jre" // CVE for 31.1
229231
force("com.fasterxml.jackson.core:jackson-core:${jacksonVersion}")
230-
force "org.eclipse.platform:org.eclipse.core.runtime:3.29.0" // CVE for < 3.29.0
232+
force "org.eclipse.platform:org.eclipse.core.runtime:3.29.0" // CVE for < 3.29.0
231233
}
232234
}
233235

@@ -254,7 +256,7 @@ publishing {
254256
}
255257
}
256258
}
257-
259+
258260
repositories {
259261
maven {
260262
name = "Snapshots"
@@ -397,8 +399,29 @@ integTest {
397399
}
398400
}
399401

400-
testClusters.integTest {
401-
testDistribution = "ARCHIVE"
402+
403+
404+
testClusters {
405+
leaderCluster {
406+
testDistribution = "ARCHIVE"
407+
setting 'path.repo', file("${buildDir}/leaderCluster/repo").absolutePath
408+
setting 'discovery.type', 'single-node'
409+
// Add custom attribute to identify this cluster
410+
setting 'node.attr.cluster_role', 'leader'
411+
configureClusterPlugins(delegate, _numNodes)
412+
}
413+
414+
followCluster {
415+
testDistribution = "ARCHIVE"
416+
setting 'path.repo', file("${buildDir}/followCluster/repo").absolutePath
417+
setting 'discovery.type', 'single-node'
418+
// Add custom attribute to identify this cluster
419+
setting 'node.attr.cluster_role', 'follower'
420+
configureClusterPlugins(delegate, _numNodes)
421+
}
422+
}
423+
424+
def configureClusterPlugins(cluster, _numNodes) {
402425
// Cluster shrink exception thrown if we try to set numberOfNodes to 1, so only apply if > 1
403426
if (_numNodes > 1) numberOfNodes = _numNodes
404427
// When running integration tests it doesn't forward the --debug-jvm to the cluster anymore
@@ -411,42 +434,50 @@ testClusters.integTest {
411434
debugPort += 1
412435
}
413436
}
414-
plugin(project.tasks.bundlePlugin.archiveFile)
415-
plugin(provider(new Callable<RegularFile>(){
416-
@Override
417-
RegularFile call() throws Exception {
418-
return new RegularFile() {
419-
@Override
420-
File getAsFile() {
421-
return configurations.zipArchive.asFileTree.getSingleFile()
437+
cluster.with {
438+
plugin(project.tasks.bundlePlugin.archiveFile)
439+
plugin(provider(new Callable<RegularFile>(){
440+
@Override
441+
RegularFile call() throws Exception {
442+
return new RegularFile() {
443+
@Override
444+
File getAsFile() {
445+
return configurations.zipArchive.asFileTree.getSingleFile()
446+
}
422447
}
423448
}
449+
}))
450+
451+
// As of ES 7.7.0 the opendistro-anomaly-detection plugin is being added to the list of plugins for the testCluster during build before
452+
// the opensearch-job-scheduler plugin, which is causing build failures. From the stack trace, this looks like a bug.
453+
//
454+
// Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [opensearch-job-scheduler], dependency of [opendistro-anomaly-detection]
455+
// at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:452)
456+
//
457+
// One explanation is that ES build script sort plugins according to the natural ordering of their names.
458+
// opendistro-anomaly-detection comes before opensearch-job-scheduler.
459+
//
460+
// The following is a comparison of different plugin installation order:
461+
// Before 7.7:
462+
// ./bin/elasticsearch-plugin install --batch file:opendistro-anomaly-detection.zip file:opensearch-job-scheduler.zip
463+
//
464+
// After 7.7:
465+
// ./bin/elasticsearch-plugin install --batch file:opensearch-job-scheduler.zip file:opendistro-anomaly-detection.zip
466+
//
467+
// A temporary hack is to reorder the plugins list after evaluation but prior to task execution when the plugins are installed.
468+
nodes.each { node ->
469+
def plugins = node.plugins
470+
def firstPlugin = plugins.get(0)
471+
plugins.remove(0)
472+
plugins.add(firstPlugin)
424473
}
425-
}))
426-
427-
// As of ES 7.7.0 the opendistro-anomaly-detection plugin is being added to the list of plugins for the testCluster during build before
428-
// the opensearch-job-scheduler plugin, which is causing build failures. From the stack trace, this looks like a bug.
429-
//
430-
// Exception in thread "main" java.lang.IllegalArgumentException: Missing plugin [opensearch-job-scheduler], dependency of [opendistro-anomaly-detection]
431-
// at org.opensearch.plugins.PluginsService.addSortedBundle(PluginsService.java:452)
432-
//
433-
// One explanation is that ES build script sort plugins according to the natural ordering of their names.
434-
// opendistro-anomaly-detection comes before opensearch-job-scheduler.
435-
//
436-
// The following is a comparison of different plugin installation order:
437-
// Before 7.7:
438-
// ./bin/elasticsearch-plugin install --batch file:opendistro-anomaly-detection.zip file:opensearch-job-scheduler.zip
439-
//
440-
// After 7.7:
441-
// ./bin/elasticsearch-plugin install --batch file:opensearch-job-scheduler.zip file:opendistro-anomaly-detection.zip
442-
//
443-
// A temporary hack is to reorder the plugins list after evaluation but prior to task execution when the plugins are installed.
444-
nodes.each { node ->
445-
def plugins = node.plugins
446-
def firstPlugin = plugins.get(0)
447-
plugins.remove(0)
448-
plugins.add(firstPlugin)
449474
}
475+
476+
}
477+
478+
testClusters.integTest {
479+
testDistribution = "ARCHIVE"
480+
configureClusterPlugins(delegate, _numNodes)
450481
}
451482

452483
task integTestRemote(type: RestIntegTestTask) {
@@ -475,7 +506,7 @@ task integTestRemote(type: RestIntegTestTask) {
475506
}
476507
}
477508

478-
2.times {i ->
509+
2.times {i ->
479510
testClusters {
480511
"${baseName}$i" {
481512
testDistribution = "ARCHIVE"
@@ -548,8 +579,8 @@ List<Provider<RegularFile>> plugins = [
548579
})
549580
]
550581

551-
// Creates 2 test clusters with 3 nodes of the old version.
552-
2.times {i ->
582+
// Creates 2 test clusters with 3 nodes of the old version.
583+
2.times {i ->
553584
task "${baseName}#oldVersionClusterTask$i"(type: RestIntegTestTask) {
554585
useCluster testClusters."${baseName}$i"
555586
filter {
@@ -560,10 +591,10 @@ List<Provider<RegularFile>> plugins = [
560591
systemProperty 'tests.plugin_bwc_version', bwcVersion
561592
nonInputProperties.systemProperty('tests.rest.cluster', "${-> testClusters."${baseName}$i".allHttpSocketURI.join(",")}")
562593
nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}$i".getName()}")
563-
}
594+
}
564595
}
565596

566-
// Upgrades one node of the old cluster to new OpenSearch version with upgraded plugin version
597+
// Upgrades one node of the old cluster to new OpenSearch version with upgraded plugin version
567598
// This results in a mixed cluster with 2 nodes on the old version and 1 upgraded node.
568599
// This is also used as a one third upgraded cluster for a rolling upgrade.
569600
task "${baseName}#mixedClusterTask"(type: RestIntegTestTask) {
@@ -621,7 +652,7 @@ task "${baseName}#rollingUpgradeClusterTask"(type: RestIntegTestTask) {
621652
nonInputProperties.systemProperty('tests.clustername', "${-> testClusters."${baseName}0".getName()}")
622653
}
623654

624-
// Upgrades all the nodes of the old cluster to new OpenSearch version with upgraded plugin version
655+
// Upgrades all the nodes of the old cluster to new OpenSearch version with upgraded plugin version
625656
// at the same time resulting in a fully upgraded cluster.
626657
task "${baseName}#fullRestartClusterTask"(type: RestIntegTestTask) {
627658
dependsOn "${baseName}#oldVersionClusterTask1"
@@ -649,16 +680,54 @@ task bwcTestSuite(type: RestIntegTestTask) {
649680

650681
run {
651682
doFirst {
652-
// There seems to be an issue when running multi node run or integ tasks with unicast_hosts
653-
// not being written, the waitForAllConditions ensures it's written
654683
getClusters().forEach { cluster ->
655684
cluster.waitForAllConditions()
656685
}
686+
687+
if (project.hasProperty('dualCluster')) {
688+
def clusterPorts = [:] // Map to store leader/follow cluster info
689+
690+
getClusters().forEach { cluster ->
691+
def clusterName = cluster.getName()
692+
def allTransportSocketURI = cluster.nodes
693+
.collectMany { node -> node.getAllTransportPortURI() }
694+
.join(",")
695+
696+
def allHttpSocketURI = cluster.nodes
697+
.collectMany { node -> node.getAllHttpSocketURI() }
698+
.join(",")
699+
700+
clusterPorts[clusterName] = [
701+
httpHosts: allHttpSocketURI,
702+
transportHosts: allTransportSocketURI
703+
]
704+
705+
println "${clusterName.capitalize()} cluster running at HTTP: ${allHttpSocketURI}, Transport: ${allTransportSocketURI}"
706+
}
707+
708+
// Pass cluster info to tests
709+
if (clusterPorts.containsKey('leader') && clusterPorts.containsKey('follow')) {
710+
systemProperty "tests.leader.cluster.name", "leader"
711+
systemProperty "tests.follow.cluster.name", "follower"
712+
713+
systemProperty "tests.cluster.leader.http_hosts", clusterPorts['leader'].httpHosts
714+
systemProperty "tests.cluster.follow.http_hosts", clusterPorts['follow'].httpHosts
715+
716+
systemProperty "tests.cluster.leader.transport_hosts", clusterPorts['leader'].transportHosts
717+
systemProperty "tests.cluster.follow.transport_hosts", clusterPorts['follow'].transportHosts
718+
}
719+
}
657720
}
658721

659-
useCluster testClusters.integTest
722+
if (project.hasProperty('dualCluster')) {
723+
useCluster testClusters.leaderCluster
724+
useCluster testClusters.followCluster
725+
} else {
726+
useCluster testClusters.integTest
727+
}
660728
}
661729

730+
662731
evaluationDependsOnChildren()
663732

664733
task release(type: Copy, group: 'build') {

0 commit comments

Comments
 (0)