Skip to content

Commit c0c315f

Browse files
authored
Update flint version to 1.0.0 in main branch (#1071)
Signed-off-by: Tomoyuki Morita <[email protected]>
1 parent b9a0dc5 commit c0c315f

File tree

9 files changed

+16
-14
lines changed

9 files changed

+16
-14
lines changed

.github/pull_request_template.md

+1
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ _List any issues this PR will resolve, e.g. Resolves [...]._
1010
- [ ] Implemented tests for combination with other commands
1111
- [ ] New added source code should include a copyright header
1212
- [ ] Commits are signed per the DCO using `--signoff`
13+
- [ ] Add `backport 0.x` label if it is a stable change which won't break existing feature
1314

1415
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
1516
For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/sql/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ sbt clean standaloneCosmetic/publishM2
6565
```
6666
then add org.opensearch:opensearch-spark-standalone_2.12 when run spark application, for example,
6767
```
68-
bin/spark-shell --packages "org.opensearch:opensearch-spark-standalone_2.12:0.7.0-SNAPSHOT" \
68+
bin/spark-shell --packages "org.opensearch:opensearch-spark-standalone_2.12:1.0.0-SNAPSHOT" \
6969
--conf "spark.sql.extensions=org.opensearch.flint.spark.FlintSparkExtensions" \
7070
--conf "spark.sql.catalog.dev=org.apache.spark.opensearch.catalog.OpenSearchCatalog"
7171
```
@@ -81,7 +81,7 @@ sbt clean sparkPPLCosmetic/publishM2
8181
Then add org.opensearch:opensearch-spark-ppl_2.12 when run spark application, for example,
8282

8383
```
84-
bin/spark-shell --packages "org.opensearch:opensearch-spark-ppl_2.12:0.7.0-SNAPSHOT" \
84+
bin/spark-shell --packages "org.opensearch:opensearch-spark-ppl_2.12:1.0.0-SNAPSHOT" \
8585
--conf "spark.sql.extensions=org.opensearch.flint.spark.FlintPPLSparkExtensions" \
8686
--conf "spark.sql.catalog.dev=org.apache.spark.opensearch.catalog.OpenSearchCatalog"
8787

build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ val sparkMinorVersion = sparkVersion.split("\\.").take(2).mkString(".")
2121

2222
ThisBuild / organization := "org.opensearch"
2323

24-
ThisBuild / version := "0.7.0-SNAPSHOT"
24+
ThisBuild / version := "1.0.0-SNAPSHOT"
2525

2626
ThisBuild / scalaVersion := scala212
2727

docker/apache-spark-sample/.env

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
MASTER_UI_PORT=8080
22
MASTER_PORT=7077
33
UI_PORT=4040
4-
PPL_JAR=../../ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-0.7.0-SNAPSHOT.jar
4+
PPL_JAR=../../ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-1.0.0-SNAPSHOT.jar

docker/integ-test/.env

+3-3
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ MASTER_UI_PORT=8080
55
MASTER_PORT=7077
66
UI_PORT=4040
77
SPARK_CONNECT_PORT=15002
8-
PPL_JAR=./ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-0.7.0-SNAPSHOT.jar
9-
FLINT_JAR=./flint-spark-integration/target/scala-2.12/flint-spark-integration-assembly-0.7.0-SNAPSHOT.jar
10-
SQL_APP_JAR=./spark-sql-application/target/scala-2.12/sql-job-assembly-0.7.0-SNAPSHOT.jar
8+
PPL_JAR=./ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-1.0.0-SNAPSHOT.jar
9+
FLINT_JAR=./flint-spark-integration/target/scala-2.12/flint-spark-integration-assembly-1.0.0-SNAPSHOT.jar
10+
SQL_APP_JAR=./spark-sql-application/target/scala-2.12/sql-job-assembly-1.0.0-SNAPSHOT.jar
1111
OPENSEARCH_NODE_MEMORY=512m
1212
OPENSEARCH_ADMIN_PASSWORD=C0rrecthorsebatterystaple.
1313
OPENSEARCH_PORT=9200

docker/spark-emr-sample/.env

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
PPL_JAR=../../ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-0.7.0-SNAPSHOT.jar
1+
PPL_JAR=../../ppl-spark-integration/target/scala-2.12/ppl-spark-integration-assembly-1.0.0-SNAPSHOT.jar

docs/index.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Currently, Flint metadata is only static configuration without version control a
6060

6161
```json
6262
{
63-
"version": "0.7.0",
63+
"version": "1.0.0",
6464
"name": "...",
6565
"kind": "skipping",
6666
"source": "...",
@@ -711,7 +711,7 @@ For now, only single or conjunct conditions (conditions connected by AND) in WHE
711711
### AWS EMR Spark Integration - Using execution role
712712
Flint use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). When running in EMR Spark, Flint use executionRole credentials
713713
```
714-
--conf spark.jars.packages=org.opensearch:opensearch-spark-standalone_2.12:0.7.0-SNAPSHOT \
714+
--conf spark.jars.packages=org.opensearch:opensearch-spark-standalone_2.12:1.0.0-SNAPSHOT \
715715
--conf spark.jars.repositories=https://aws.oss.sonatype.org/content/repositories/snapshots \
716716
--conf spark.emr-serverless.driverEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 \
717717
--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 \
@@ -753,7 +753,7 @@ Flint use [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJa
753753
```
754754
3. Set the spark.datasource.flint.customAWSCredentialsProvider property with value as com.amazonaws.emr.AssumeRoleAWSCredentialsProvider. Set the environment variable ASSUME_ROLE_CREDENTIALS_ROLE_ARN with the ARN value of CrossAccountRoleB.
755755
```
756-
--conf spark.jars.packages=org.opensearch:opensearch-spark-standalone_2.12:0.7.0-SNAPSHOT \
756+
--conf spark.jars.packages=org.opensearch:opensearch-spark-standalone_2.12:1.0.0-SNAPSHOT \
757757
--conf spark.jars.repositories=https://aws.oss.sonatype.org/content/repositories/snapshots \
758758
--conf spark.emr-serverless.driverEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 \
759759
--conf spark.executorEnv.JAVA_HOME=/usr/lib/jvm/java-17-amazon-corretto.x86_64 \

docs/ppl-lang/PPL-on-Spark.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ sbt clean sparkPPLCosmetic/publishM2
3434
```
3535
then add org.opensearch:opensearch-spark_2.12 when run spark application, for example,
3636
```
37-
bin/spark-shell --packages "org.opensearch:opensearch-spark-ppl_2.12:0.7.0-SNAPSHOT"
37+
bin/spark-shell --packages "org.opensearch:opensearch-spark-ppl_2.12:1.0.0-SNAPSHOT"
3838
```
3939

4040
### PPL Extension Usage
@@ -46,7 +46,7 @@ spark-sql --conf "spark.sql.extensions=org.opensearch.flint.spark.FlintPPLSparkE
4646
```
4747

4848
### Running With both Flint & PPL Extensions
49-
In order to make use of both flint and ppl extension, one can simply add both jars (`org.opensearch:opensearch-spark-ppl_2.12:0.7.0-SNAPSHOT`,`org.opensearch:opensearch-spark_2.12:0.7.0-SNAPSHOT`) to the cluster's
49+
In order to make use of both flint and ppl extension, one can simply add both jars (`org.opensearch:opensearch-spark-ppl_2.12:1.0.0-SNAPSHOT`,`org.opensearch:opensearch-spark_2.12:1.0.0-SNAPSHOT`) to the cluster's
5050
classpath.
5151

5252
Next need to configure both extensions :

flint-commons/src/main/scala/org/opensearch/flint/common/FlintVersion.scala

+2-1
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ object FlintVersion {
2121
val V_0_5_0: FlintVersion = FlintVersion("0.5.0")
2222
val V_0_6_0: FlintVersion = FlintVersion("0.6.0")
2323
val V_0_7_0: FlintVersion = FlintVersion("0.7.0")
24+
val V_1_0_0: FlintVersion = FlintVersion("1.0.0")
2425

25-
def current(): FlintVersion = V_0_7_0
26+
def current(): FlintVersion = V_1_0_0
2627
}

0 commit comments

Comments
 (0)