More robust sample testing nodejs #427
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| #!/bin/bash | ||
|
|
||
| aws kinesis delete-stream --stream-name $STREAM_NAME || true | ||
|
|
||
| # Reset the values of checkpoint, leaseCounter, ownerSwitchesSinceCheckpoint, and leaseOwner in DynamoDB table | ||
| echo "Resetting DDB table" | ||
| aws dynamodb update-item \ | ||
| --table-name $APP_NAME \ | ||
| --key '{"leaseKey": {"S": "shardId-000000000000"}}' \ | ||
| --update-expression "SET checkpoint = :checkpoint, leaseCounter = :counter, ownerSwitchesSinceCheckpoint = :switches, leaseOwner = :owner" \ | ||
| --expression-attribute-values '{ | ||
| ":checkpoint": {"S": "TRIM_HORIZON"}, | ||
| ":counter": {"N": "0"}, | ||
| ":switches": {"N": "0"}, | ||
| ":owner": {"S": "AVAILABLE"} | ||
| }' \ | ||
| --return-values NONE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| for i in {1..10}; do | ||
| if aws kinesis create-stream --stream-name $STREAM_NAME --shard-count 1; then | ||
| break | ||
| else | ||
| echo "Stream creation failed, attempt $i/10. Waiting $((i * 3)) seconds..." | ||
| sleep $((i * 3)) | ||
| fi | ||
| done | ||
| aws kinesis wait stream-exists --stream-name $STREAM_NAME |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| # Manipulate sample.properties file that the KCL application pulls properties from (ex: streamName, applicationName) | ||
| # Depending on the OS, different properties need to be changed | ||
| if [[ "$RUNNER_OS" == "macOS" ]]; then | ||
| sed -i "" "s/streamName = .*/streamName = $STREAM_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "" "s/applicationName = .*/applicationName = $APP_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "" "s/#*idleTimeBetweenReadsInMillis.*/idleTimeBetweenReadsInMillis = 250/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "" "s/stream : 'kclnodejssample'/stream : '$STREAM_NAME'/" samples/basic_sample/producer/config.js | ||
| sed -i "" "s/shards : 2/shards : 1/" samples/basic_sample/producer/config.js | ||
| elif [[ "$RUNNER_OS" == "Linux" ]]; then | ||
| sed -i "s/streamName = .*/streamName = $STREAM_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/applicationName = .*/applicationName = $APP_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/#*idleTimeBetweenReadsInMillis.*/idleTimeBetweenReadsInMillis = 250/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/stream : 'kclnodejssample'/stream : '$STREAM_NAME'/" samples/basic_sample/producer/config.js | ||
| sed -i "s/shards : 2/shards : 1/" samples/basic_sample/producer/config.js | ||
| elif [[ "$RUNNER_OS" == "Windows" ]]; then | ||
| sed -i "s/streamName = .*/streamName = $STREAM_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/applicationName = .*/applicationName = $APP_NAME/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/#*idleTimeBetweenReadsInMillis.*/idleTimeBetweenReadsInMillis = 250/" samples/basic_sample/consumer/sample.properties | ||
| sed -i "s/stream : 'kclnodejssample'/stream : '$STREAM_NAME'/" samples/basic_sample/producer/config.js | ||
| sed -i "s/shards : 2/shards : 1/" samples/basic_sample/producer/config.js | ||
| else | ||
| echo "Unknown OS: $RUNNER_OS" | ||
| exit 1 | ||
| fi | ||
|
|
||
| cat samples/basic_sample/consumer/sample.properties | ||
| cat samples/basic_sample/producer/config.js | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| cd samples/basic_sample/producer | ||
| node sample_kinesis_producer_app.js | ||
|
|
||
| # Get records from stream to verify they exist before continuing | ||
| SHARD_ITERATOR=$(aws kinesis get-shard-iterator --stream-name $STREAM_NAME --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --query 'ShardIterator' --output text) | ||
| INITIAL_RECORDS=$(aws kinesis get-records --shard-iterator $SHARD_ITERATOR) | ||
| RECORD_COUNT_BEFORE=$(echo $INITIAL_RECORDS | jq '.Records | length') | ||
|
|
||
| if [ "$RECORD_COUNT_BEFORE" -eq 0 ]; then | ||
| echo "No records found in stream. Test cannot proceed." | ||
| exit 1 | ||
| fi | ||
| echo "Found $RECORD_COUNT_BEFORE records in stream before KCL start" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| #!/bin/bash | ||
| set -e | ||
| set -o pipefail | ||
|
|
||
| # Reset the values of checkpoint, leaseCounter, ownerSwitchesSinceCheckpoint, and leaseOwner in DynamoDB table | ||
| echo "Resetting checkpoint for shardId-000000000000" | ||
| aws dynamodb update-item \ | ||
| --table-name $APP_NAME \ | ||
| --key '{"leaseKey": {"S": "shardId-000000000000"}}' \ | ||
| --update-expression "SET checkpoint = :checkpoint, leaseCounter = :counter, ownerSwitchesSinceCheckpoint = :switches, leaseOwner = :owner" \ | ||
| --expression-attribute-values '{ | ||
| ":checkpoint": {"S": "TRIM_HORIZON"}, | ||
| ":counter": {"N": "0"}, | ||
| ":switches": {"N": "0"}, | ||
| ":owner": {"S": "AVAILABLE"} | ||
| }' \ | ||
| --return-values NONE | ||
|
|
||
| # Get records from stream to verify they exist before continuing | ||
| SHARD_ITERATOR=$(aws kinesis get-shard-iterator --stream-name $STREAM_NAME --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --query 'ShardIterator' --output text) | ||
| INITIAL_RECORDS=$(aws kinesis get-records --shard-iterator $SHARD_ITERATOR) | ||
| RECORD_COUNT_BEFORE=$(echo $INITIAL_RECORDS | jq '.Records | length') | ||
|
|
||
| echo "Found $RECORD_COUNT_BEFORE records in stream before KCL start" | ||
|
|
||
| if [[ "$RUNNER_OS" == "macOS" ]]; then | ||
| brew install coreutils | ||
| cd samples/basic_sample/consumer | ||
| KCL_COMMAND="../../../bin/kcl-bootstrap --java /usr/bin/java -e -p ./sample.properties" | ||
| gtimeout 300 $KCL_COMMAND 2>&1 | tee kcl_output.log || [ $? -eq 124 ] | ||
| elif [[ "$RUNNER_OS" == "Linux" ]]; then | ||
| cd samples/basic_sample/consumer | ||
| KCL_COMMAND="../../../bin/kcl-bootstrap -e -p ./sample.properties" | ||
| timeout 300 $KCL_COMMAND 2>&1 | tee kcl_output.log || [ $? -eq 124 ] | ||
| elif [[ "$RUNNER_OS" == "Windows" ]]; then | ||
| cd samples/basic_sample/consumer | ||
| KCL_COMMAND="../../../bin/kcl-bootstrap -e -p ./sample.properties" | ||
| timeout 300 $KCL_COMMAND 2>&1 | tee kcl_output.log || [ $? -eq 124 ] | ||
|
Comment on lines
+31
to
+38
There was a problem hiding this comment. These look to be the same commands, can we put an |
||
| else | ||
| echo "Unknown OS: $RUNNER_OS" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "---------ERROR LOGS HERE-------" | ||
| grep -i error kcl_output.log || echo "No errors found in logs" | ||
|
|
||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| LEASE_EXISTS=$(aws dynamodb scan --table-name $APP_NAME --select "COUNT" --query "Count" --output text || echo "0") | ||
| CHECKPOINT_EXISTS=$(aws dynamodb scan --table-name $APP_NAME --select "COUNT" --filter-expression "attribute_exists(checkpoint) AND checkpoint <> :trim_horizon" --expression-attribute-values '{":trim_horizon": {"S": "TRIM_HORIZON"}}' --query "Count" --output text || echo "0") | ||
|
|
||
| echo "Found $LEASE_EXISTS leases and $CHECKPOINT_EXISTS non-TRIM-HORIZON checkpoints in DynamoDB" | ||
|
|
||
| echo "Printing checkpoint values" | ||
| aws dynamodb scan --table-name $APP_NAME --projection-expression "leaseKey,checkpoint" --output json | ||
|
|
||
| if [ "$LEASE_EXISTS" -gt 0 ] && [ "$CHECKPOINT_EXISTS" -gt 0 ]; then | ||
| echo "Test passed: Found both leases and non-TRIM_HORIZON checkpoints in DDB (KCL is fully functional)" | ||
| exit 0 | ||
| else | ||
| echo "Test failed: KCL not fully functional" | ||
| echo "Lease(s) found: $LEASE_EXISTS" | ||
| echo "non-TRIM_HORIZON checkpoint(s) found: $CHECKPOINT_EXISTS" | ||
| exit 1 | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these all the same commands? Can we merge them all under one if statement so we don't repeat code?