Skip to content

Commit ba1bcd7

Browse files
jbrinkmanjamesx-improvingalexr-bqalexey-temnikovprateek-kumar-improving
authored
[BACKPORT 2.1] Merge Release 2.1 back into Main (#4753)
* Go: Add MUSL support (#4476) * Go: Switch to MUSL binary Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> * rustup add musl targets Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> * lint Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> * rustup add musl targets in Makefile Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> * Add pathing Signed-off-by: Alex Rehnby-Martin <[email protected]> * Trigger CodeQL on main Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update CI Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix typo Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix typo Signed-off-by: Alex Rehnby-Martin <[email protected]> * Modify CD Signed-off-by: Alex Rehnby-Martin <[email protected]> * Modify CD Signed-off-by: Alex Rehnby-Martin <[email protected]> * Modify CD Signed-off-by: Alex Rehnby-Martin <[email protected]> * Modify CD Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update docs Signed-off-by: Alex Rehnby-Martin <[email protected]> * Format Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix Signed-off-by: Alex Rehnby-Martin <[email protected]> * Address feedback Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update doc Signed-off-by: Alex Rehnby-Martin <[email protected]> --------- Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> Co-authored-by: Alex Rehnby-Martin <[email protected]> Co-authored-by: Alexey Temnikov <[email protected]> * Java: Migration guide for Jedis compatibility layer (#4672) (#4681) * Java: Migration guide for Jedis compatibility layer Signed-off-by: Prateek Kumar <[email protected]> * [Backport to 2.1] Python: Add Multi-Database Support for Cluster Mode Valkey 9.0 (#4694) Python: Add Multi-Database Support for Cluster Mode Valkey 9.0 (#4659) * - Implement database selection for cluster clients when database_id != 0 - Send SELECT command to all cluster nodes using MultiNode routing - Add comprehensive error handling with clear error messages - Include test for database selection error handling in cluster mode - Support backward compatibility with servers that don't support multi-db cluster mode - Enable cluster-databases=16 for Valkey 9.0+ in cluster manager This enables cluster clients to work with non-default databases on Valkey 9.0+ servers configured with cluster-databases support, while gracefully handling older servers that don't support this feature. * update valkey9 multi db tests * fix lint * fixing valkey 9 cluster tests * add to route to all nodes in the core * rust lint fix * implement database selection in cluster mode via connection parameters - Add database_id field to ClusterParams and BuilderParams structs - Add database() method to ClusterClientBuilder for setting database ID - Pass database_id through connection info instead of post-connection SELECT - Remove SELECT command from cluster routing - Remove post-connection SELECT command logic from create_cluster_client - Add comprehensive tests for database isolation and reconnection behavior - Verify database ID persistence across node reconnections - Test multi-database cluster support with proper isolation verification This change moves database selection from a post-connection SELECT command to a connection parameter, which is more reliable for cluster mode and handles reconnections automatically. The approach works with servers that support multi-database cluster configurations (like Valkey 9.0+). * add valkey9 constraint in the core tests * fix core test not skipping test when version was lower than valkey 9 * Fix version check * refactored test test_set_database_id_after_reconnection to be similar from standalone removed is_valkey_9_or_higher * removed tests and cleanup * renamed builder.database to builder.database_id * python: Add multi-database support for cluster and standalone modes - Add database_id parameter to BaseClientConfiguration, GlideClientConfiguration, and GlideClusterClientConfiguration - Implement SELECT command for both cluster and standalone modes with comprehensive documentation - Add validation for database_id parameter (0-15 range, integer type) - Refactor configuration protobuf request creation with helper methods for better maintainability - Add extensive test coverage for database_id configuration and validation - Include production warnings and recommended approaches in SELECT command documentation - Support routing configuration for cluster mode SELECT operations - Ensure database_id persists across reconnections when set in client configuration Documentation: - Added comprehensive docstrings with warnings about SELECT command limitations - Included examples showing recommended database_id configuration approach - Documented reconnection behavior and cluster-specific routing requirements * fixing database_id restriction * fixed test test_standalone_client_with_very_high_database_ids * fix python tests with higher databases id * fix lint error * refactoring out the route option from the select command * Remove SELECT command from cluster mode and enable cluster tests - Remove select() method from ClusterCommands in both async and sync clients - Enable cluster mode testing for database_id tests (parametrize True, False) - Delete database_id integration test file The SELECT command is not recommended for cluster mode due to reconnection behavior where nodes revert to configured database_id, not the selected one. * removed database_id validation| renamed tests to be cluster/standalone agnostic clean up comments * added changelog * lint fix * fix test database_id should be skipped for versions lower than 9.0.0 * fix test * added select rounting refactored extract_client_id to be in utilities * fix database_id documentation added custom command test using select --------- Signed-off-by: affonsov <[email protected]> * [Backport 2.1] Go: Add Multi-Database Support for Cluster Mode Valkey 9.0 (#4696) Go: Add Multi-Database Support for Cluster Mode Valkey 9.0 (#4660) * - Implement database selection for cluster clients when database_id != 0 - Send SELECT command to all cluster nodes using MultiNode routing - Add comprehensive error handling with clear error messages - Include test for database selection error handling in cluster mode - Support backward compatibility with servers that don't support multi-db cluster mode - Enable cluster-databases=16 for Valkey 9.0+ in cluster manager This enables cluster clients to work with non-default databases on Valkey 9.0+ servers configured with cluster-databases support, while gracefully handling older servers that don't support this feature. * update valkey9 multi db tests * fix lint * fixing valkey 9 cluster tests * add to route to all nodes in the core * rust lint fix * implement database selection in cluster mode via connection parameters - Add database_id field to ClusterParams and BuilderParams structs - Add database() method to ClusterClientBuilder for setting database ID - Pass database_id through connection info instead of post-connection SELECT - Remove SELECT command from cluster routing - Remove post-connection SELECT command logic from create_cluster_client - Add comprehensive tests for database isolation and reconnection behavior - Verify database ID persistence across node reconnections - Test multi-database cluster support with proper isolation verification This change moves database selection from a post-connection SELECT command to a connection parameter, which is more reliable for cluster mode and handles reconnections automatically. The approach works with servers that support multi-database cluster configurations (like Valkey 9.0+). * add valkey9 constraint in the core tests * fix core test not skipping test when version was lower than valkey 9 * Fix version check * refactored test test_set_database_id_after_reconnection to be similar from standalone removed is_valkey_9_or_higher * removed tests and cleanup * renamed builder.database to builder.database_id * go: Add multi-database support for cluster mode clients - Add DatabaseId field to baseClientConfiguration for both standalone and cluster clients - Implement WithDatabaseId() method for ClusterClientConfiguration - Add database ID validation with proper error handling for negative values - Refactor standalone client to use shared database ID logic from base configuration - Add Select() and SelectWithOptions() methods for cluster clients with routing support - Include comprehensive test coverage for database isolation, reconnection persistence, and error handling - Add backward compatibility support - clients default to database 0 when no database_id specified - Add server version compatibility checks (cluster multi-database requires Valkey 9.0+) - Update documentation with warnings about SELECT command limitations and recommended configuration approach This enables cluster mode clients to connect to specific databases at initialization time, with the database selection persisting across reconnections, unlike the SELECT command which resets on reconnection. * fixed go tests - batch tests - database id tests * removed selectWithOptions * go: Remove Select command from cluster clients - Remove Select method from ClusterClient and related interfaces - Delete comprehensive database_id integration tests - Remove Select command from batch operations - Remove Select examples and tests - Add example for cluster client with database_id configuration The Select command is being removed in favor of using the database_id configuration parameter, which provides persistent database selection across reconnections. * remove config validation simplified tests * added changelog * removing batch test, and readding standalone select and removing batch select --------- Signed-off-by: affonsov <[email protected]> * [Backport 2.1] Java: Multi-Database Support for Cluster Mode Valkey 9.0 (#4697) Java: Multi-Database Support for Cluster Mode Valkey 9.0 (#4658) * - Implement database selection for cluster clients when database_id != 0 - Send SELECT command to all cluster nodes using MultiNode routing - Add comprehensive error handling with clear error messages - Include test for database selection error handling in cluster mode - Support backward compatibility with servers that don't support multi-db cluster mode - Enable cluster-databases=16 for Valkey 9.0+ in cluster manager This enables cluster clients to work with non-default databases on Valkey 9.0+ servers configured with cluster-databases support, while gracefully handling older servers that don't support this feature. * update valkey9 multi db tests * fix lint * fixing valkey 9 cluster tests * add to route to all nodes in the core * rust lint fix * implement database selection in cluster mode via connection parameters - Add database_id field to ClusterParams and BuilderParams structs - Add database() method to ClusterClientBuilder for setting database ID - Pass database_id through connection info instead of post-connection SELECT - Remove SELECT command from cluster routing - Remove post-connection SELECT command logic from create_cluster_client - Add comprehensive tests for database isolation and reconnection behavior - Verify database ID persistence across node reconnections - Test multi-database cluster support with proper isolation verification This change moves database selection from a post-connection SELECT command to a connection parameter, which is more reliable for cluster mode and handles reconnections automatically. The approach works with servers that support multi-database cluster configurations (like Valkey 9.0+). * add valkey9 constraint in the core tests * fix core test not skipping test when version was lower than valkey 9 * Fix version check * refactored test test_set_database_id_after_reconnection to be similar from standalone removed is_valkey_9_or_higher * removed tests and cleanup * renamed builder.database to builder.database_id * Java: implement multi-database support for cluster mode - Move databaseId from GlideClientConfiguration to BaseClientConfiguration - Add databaseId validation and protobuf integration - Update SELECT command with AllNodes routing and reconnection warnings - Add connection management for database selection after auth - Add comprehensive tests with Valkey 9+ version guards Completes Java client implementation for Valkey 9 cluster multi-database support. * fixing database id restriction * fixed java test error select_command_invalid_database_standalone * fixed lint errors * Refactored select command to route to all nodes per default Removed the option to provide a route * java lint fix * java: Remove SELECT command from cluster client - Remove select() method from GlideClusterClient and ConnectionManagementClusterCommands interface - Delete MultiDatabaseTests.java which tested SELECT command functionality - Remove SELECT command tests from SharedCommandTests.java - Add focused test for cluster database_id configuration in ClusterClientTests - Remove SELECT import from GlideClusterClient The SELECT command is being removed from the cluster client API as database selection should be handled through client configuration (databaseId parameter) rather than runtime commands, which provides better consistency and avoids reconnection issues. * remove database_id validatation and simplified tests * added changelog * fix documentation * fix databaseId documentation --------- Signed-off-by: affonsov <[email protected]> * [Backport 2.1] Node: Add Multi-Database Support for Cluster Mode (Valkey 9.0) (#4698) Node: Add Multi-Database Support for Cluster Mode (Valkey 9.0) (#4657) * - Implement database selection for cluster clients when database_id != 0 - Send SELECT command to all cluster nodes using MultiNode routing - Add comprehensive error handling with clear error messages - Include test for database selection error handling in cluster mode - Support backward compatibility with servers that don't support multi-db cluster mode - Enable cluster-databases=16 for Valkey 9.0+ in cluster manager This enables cluster clients to work with non-default databases on Valkey 9.0+ servers configured with cluster-databases support, while gracefully handling older servers that don't support this feature. * update valkey9 multi db tests * fix lint * fixing valkey 9 cluster tests * add to route to all nodes in the core * rust lint fix * implement database selection in cluster mode via connection parameters - Add database_id field to ClusterParams and BuilderParams structs - Add database() method to ClusterClientBuilder for setting database ID - Pass database_id through connection info instead of post-connection SELECT - Remove SELECT command from cluster routing - Remove post-connection SELECT command logic from create_cluster_client - Add comprehensive tests for database isolation and reconnection behavior - Verify database ID persistence across node reconnections - Test multi-database cluster support with proper isolation verification This change moves database selection from a post-connection SELECT command to a connection parameter, which is more reliable for cluster mode and handles reconnections automatically. The approach works with servers that support multi-database cluster configurations (like Valkey 9.0+). * add valkey9 constraint in the core tests * fix core test not skipping test when version was lower than valkey 9 * Fix version check * refactored test test_set_database_id_after_reconnection to be similar from standalone removed is_valkey_9_or_higher * removed tests and cleanup * renamed builder.database to builder.database_id * node changes * move tests to sharedtests.ts * reafactoring to select to all nodes being in the core * node: Remove SELECT command support from cluster client - Remove select() method from GlideClusterClient - Remove createSelect import from cluster client - Remove extensive test coverage for SELECT functionality - Remove database ID validation tests from client internals - Add minimal database ID test for cluster client configuration - Clean up ConfigurationError import that's no longer needed This change removes the SELECT command implementation that was added for Valkey 9.0+ cluster support, likely due to reliability concerns with database switching in cluster mode or to simplify the API. * added changelog * fix test, and removed comment --------- Signed-off-by: affonsov <[email protected]> * [Backport 2.1] Python sync: removed select command (#4705) Python sync: removed select command (#4684) removed sync select command Signed-off-by: Lior Sventitzky <[email protected]> * [Backport 2.1] CI/CD: Added self hosted macOS ARM runners (#4706) * CI: Added self hosted macOS runners (#4683) added self hosted runners for macos Signed-off-by: Lior Sventitzky <[email protected]> * CD: removed self hosted mac arm runners from required CD platforms #4700 (#4702) removed self hosted mac from cd, reverted npm-cd Signed-off-by: Lior Sventitzky <[email protected]> --------- Signed-off-by: Lior Sventitzky <[email protected]> * [backport 2.1] Python: fixed pypi-cd workflow, fixed release tests (#4708) Python: fixed pypi-cd workflow, fixed release tests (#4703) fixed pypi upload, fixed release tests Signed-off-by: Lior Sventitzky <[email protected]> * Fix FFI tests for non-alpine distros (#4711) Signed-off-by: Jeremy Parr-Pearson <[email protected]> * MUSL Java Build Fixes (#4712) * Use zigbuild for easier cross-compilation Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix Signed-off-by: Alex Rehnby-Martin <[email protected]> * Use zigbuild for easier cross-compilation Signed-off-by: Alex Rehnby-Martin <[email protected]> * Use zigbuild for easier cross-compilation Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix musl classifier / detection Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix musl classifier / detection Signed-off-by: Alex Rehnby-Martin <[email protected]> * Fix musl classifier / detection Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update docs Signed-off-by: Alex Rehnby-Martin <[email protected]> * Exclude arm musl test Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update arm runner Signed-off-by: Alex Rehnby-Martin <[email protected]> * Update go exclusion Signed-off-by: Alex Rehnby-Martin <[email protected]> * Format Signed-off-by: Alex Rehnby-Martin <[email protected]> --------- Signed-off-by: Alex Rehnby-Martin <[email protected]> * Revert go upload step change (#4716) Signed-off-by: Alex Rehnby-Martin <[email protected]> * CD: Fix Node CD delete artifact issue (#4717) Signed-off-by: James Xin <[email protected]> * CD: Fix Go musl CD issue (#4719) Signed-off-by: James Xin <[email protected]> * CD: Fix Node CD delete artifact issue 2nd try (#4718) * CD: Fix Node CD delete artifact issue 2nd try Signed-off-by: James Xin <[email protected]> * Linter Signed-off-by: James Xin <[email protected]> * remove temporary testing Signed-off-by: James Xin <[email protected]> --------- Signed-off-by: James Xin <[email protected]> * Python: Update ExpiryType enum references in hash field expiration docs (#4715) python: Update ExpiryType enum references in hash field expiration docs Replace deprecated ExpiryType enum values with new names in docstrings: - EX → SEC - PX → MILLSEC - EXAT → UNIX_SEC - PXAT → UNIX_MILLSEC - KEEPTTL → KEEP_TTL Updates documentation for hash field expiration commands in both async and sync clients. Signed-off-by: affonsov <[email protected]> * [Backport 2.1] Python sync/CD: added description to pypi package of the sync client (#4720) Python sync/CD: added description to pypi package of the sync client (#4714) added description to sync client Signed-off-by: Lior Sventitzky <[email protected]> * Java: Compatibility layer module (#4692) * Java: Compatibility layer module Signed-off-by: Prateek Kumar <[email protected]> * Java: fix integration tests Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql for java code Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix codeql java build Signed-off-by: Prateek Kumar <[email protected]> * Java: Update java cd Signed-off-by: Prateek Kumar <[email protected]> * Java: lint issues fixed Signed-off-by: Prateek Kumar <[email protected]> * Java: Add signing configuration to jedis-compatibility build.gradle Signed-off-by: Prateek Kumar <[email protected]> * Java: update dependency Signed-off-by: Prateek Kumar <[email protected]> * Java: Update valkey-glide-jedis-compatibility file path Signed-off-by: Prateek Kumar <[email protected]> * Java: Fix build.gradle publishing code Signed-off-by: Prateek Kumar <[email protected]> --------- Signed-off-by: Prateek Kumar <[email protected]> * Fix: Copy GPG secring to jedis-compatibility in Java CD workflow (#4723) fix(java-cd): copy secring.gpg to jedis-compatibility for signing Signed-off-by: jbrinkman <[email protected]> * [Backport 2.1] Python Sync: revert license format (#4732) Python Sync: revert license format (#4731) revert license format Signed-off-by: Lior Sventitzky <[email protected]> * Java: Fix Jedis compatibility layer valkey-glide dependency (#4737) * Java: Fix Jedis compatibility layer valkey-glide dependency Signed-off-by: Prateek Kumar <[email protected]> * [Backport 2.1] Python: improve UDS socket error handling (#4733) (#4755) Python: improve UDS socket error handling (#4733) fixed uds error handling Signed-off-by: Lior Sventitzky <[email protected]> Co-authored-by: Lior Sventitzky <[email protected]> --------- Signed-off-by: James Xin <[email protected]> Signed-off-by: Alex Rehnby-Martin <[email protected]> Signed-off-by: Prateek Kumar <[email protected]> Signed-off-by: affonsov <[email protected]> Signed-off-by: Lior Sventitzky <[email protected]> Signed-off-by: Jeremy Parr-Pearson <[email protected]> Signed-off-by: jbrinkman <[email protected]> Co-authored-by: James Xin <[email protected]> Co-authored-by: Alex Rehnby-Martin <[email protected]> Co-authored-by: Alexey Temnikov <[email protected]> Co-authored-by: prateek-kumar-improving <[email protected]> Co-authored-by: affonsov <[email protected]> Co-authored-by: Lior Sventitzky <[email protected]> Co-authored-by: Jeremy Parr-Pearson <[email protected]>
1 parent 52084a5 commit ba1bcd7

File tree

135 files changed

+815
-123
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

135 files changed

+815
-123
lines changed

.github/json_matrices/build-matrix.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,11 @@
5353
"ARCH": "arm64",
5454
"TARGET": "aarch64-unknown-linux-musl",
5555
"CD_TARGET": "aarch64-unknown-linux-gnu",
56-
"RUNNER": "ubuntu-latest-arm",
56+
"RUNNER": "ubuntu-24.04-arm",
5757
"CD_RUNNER": "ubuntu-24.04-arm",
5858
"IMAGE": "node:lts-alpine",
5959
"CONTAINER_OPTIONS": "--user root --privileged --rm",
60-
"PACKAGE_MANAGERS": ["npm", "maven"],
60+
"PACKAGE_MANAGERS": ["npm", "maven", "pkg_go_dev"],
6161
"languages": []
6262
},
6363
{
@@ -69,8 +69,8 @@
6969
"RUNNER": "ubuntu-latest",
7070
"IMAGE": "node:lts-alpine",
7171
"CONTAINER_OPTIONS": "--user root --privileged",
72-
"PACKAGE_MANAGERS": ["npm", "maven"],
73-
"languages": ["node", "java"]
72+
"PACKAGE_MANAGERS": ["npm", "maven", "pkg_go_dev"],
73+
"languages": ["node", "java", "go"]
7474
},
7575
{
7676
"OS": "amazon-linux",

.github/workflows/go-cd.yml

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -116,17 +116,17 @@ jobs:
116116
uses: ./.github/workflows/install-shared-dependencies
117117
with:
118118
os: ${{ matrix.host.OS }}
119-
target: ${{ matrix.host.TARGET }}
119+
target: ${{ matrix.host.CD_TARGET || matrix.host.TARGET }}
120120
github-token: ${{ secrets.GITHUB_TOKEN }}
121121
- name: Build Go client
122122
working-directory: go
123123
env:
124124
RELEASE_VERSION: ${{ needs.validate-release-version.outputs.RELEASE_VERSION }}
125125
run: |
126126
make install-build-tools
127-
make build GLIDE_VERSION="${RELEASE_VERSION}"
127+
make build GLIDE_VERSION="${RELEASE_VERSION}" TARGET_TRIPLET="${{ matrix.host.TARGET }}"
128128
- name: Move FFI artifacts on linux
129-
if: ${{ contains(matrix.host.TARGET, 'linux-gnu') }}
129+
if: ${{ contains(matrix.host.TARGET, 'linux') }}
130130
run: |
131131
mkdir -p $GITHUB_WORKSPACE/ffi/target/release
132132
cp ffi/target/*/release/libglide_ffi.a $GITHUB_WORKSPACE/ffi/target/release/
@@ -187,6 +187,12 @@ jobs:
187187
fail-fast: false
188188
matrix:
189189
host: ${{ fromJson(needs.load-platform-matrix.outputs.PLATFORM_MATRIX) }}
190+
# Excluding musl targets for testing for now as there are issues with the runners
191+
exclude:
192+
- host:
193+
TARGET: aarch64-unknown-linux-musl
194+
- host:
195+
TARGET: x86_64-unknown-linux-musl
190196
runs-on: ${{ matrix.host.RUNNER }}
191197
steps:
192198
- name: Setup self-hosted runner access

.github/workflows/go.yml

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -259,10 +259,17 @@ jobs:
259259
steps:
260260
- name: Install git
261261
run: |
262-
yum update
263-
yum install -y git tar
264-
git config --global --add safe.directory "$GITHUB_WORKSPACE"
265-
echo IMAGE=amazonlinux:latest | sed -r 's/:/-/g' >> $GITHUB_ENV
262+
if [[ "${{ matrix.host.OS }}" == "amazon-linux" ]]; then
263+
yum update
264+
yum install -y git tar
265+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
266+
echo IMAGE=amazonlinux:latest | sed -r 's/:/-/g' >> $GITHUB_ENV
267+
elif [[ "${{ matrix.host.TARGET }}" == *"musl"* ]]; then
268+
apk update
269+
apk add git bash tar
270+
git config --global --add safe.directory "$GITHUB_WORKSPACE"
271+
fi
272+
266273
# Replace `:` in the variable otherwise it can't be used in `upload-artifact`
267274
- uses: actions/checkout@v4
268275
with:
@@ -282,6 +289,19 @@ jobs:
282289
github-token: ${{ secrets.GITHUB_TOKEN }}
283290
engine-version: ${{ matrix.engine.version }}
284291

292+
- name: Install Musl Dependencies
293+
if: ${{ contains(matrix.host.TARGET, 'musl') }}
294+
run: |
295+
apk add clang lld compiler-rt
296+
# Ensure Rust is in PATH for container environments
297+
echo "$HOME/.cargo/bin:$PATH" >> $GITHUB_PATH
298+
299+
- name: Install zig
300+
if: ${{ contains(matrix.host.TARGET, 'musl') }}
301+
uses: ./.github/workflows/install-zig
302+
with:
303+
target: ${{ matrix.host.TARGET }}
304+
285305
- uses: actions/cache@v4
286306
with:
287307
path: |

.github/workflows/install-zig/action.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,14 @@ runs:
1717
run: |
1818
if [[ `cat /etc/os-release | grep "Amazon Linux"` ]]; then
1919
yum install -y python3-pip
20+
pip3 install ziglang
21+
elif [[ `cat /etc/os-release | grep "Alpine"` ]]; then
22+
apk add py3-pip
23+
pip3 install ziglang --break-system-packages
2024
else
2125
sudo apt install -y python3-pip
26+
pip3 install ziglang
2227
fi
23-
pip3 install ziglang
2428
cargo install --locked cargo-zigbuild
2529
2630
# Set environment variable to prevent cargo-zigbuild from auto-detecting malformed targets

.github/workflows/java-cd.yml

Lines changed: 49 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,9 @@ jobs:
3939
export PLATFORM_MATRIX=$(jq 'map(
4040
select(.PACKAGE_MANAGERS != null and (.PACKAGE_MANAGERS | contains(["maven"])))
4141
| .RUNNER = (
42-
if (.RUNNER | type == "array")
43-
then (.RUNNER | map(if . == "ephemeral" then "persistent" else . end))
44-
else (if .RUNNER == "ephemeral" then "persistent" else .RUNNER end)
42+
if (.RUNNER | type == "array")
43+
then (.RUNNER | map(if . == "ephemeral" then "persistent" else . end))
44+
else (if .RUNNER == "ephemeral" then "persistent" else .RUNNER end)
4545
end
4646
)
4747
)' < .github/json_matrices/build-matrix.json | jq -c .)
@@ -129,45 +129,77 @@ jobs:
129129
${{ runner.os }}-gradle-cd-
130130
${{ runner.os }}-gradle-
131131
132-
- name: Create secret key ring file
132+
- name: Create secret key ring file for all Java submodules
133133
working-directory: java/client
134134
run: |
135+
# Decode the provided base64 GPG key into the client module
135136
echo "$SECRING_GPG" | base64 --decode > ./secring.gpg
136-
ls -ltr
137+
138+
# Copy the key ring file into the jedis-compatibility module which also performs signing
139+
if [ -d ../jedis-compatibility ]; then
140+
cp ./secring.gpg ../jedis-compatibility/secring.gpg
141+
else
142+
echo "jedis-compatibility module directory not found" >&2
143+
exit 1
144+
fi
145+
146+
echo "Listing key ring files to verify presence:"
147+
ls -l ./secring.gpg ../jedis-compatibility/secring.gpg
137148
env:
138149
SECRING_GPG: ${{ secrets.SECRING_GPG }}
139150

140151
- name: Build java client
141152
working-directory: java
142153
run: |
143154
if [[ "${{ matrix.host.TARGET }}" == *"musl"* ]]; then
155+
# Build and publish client first
144156
./gradlew --build-cache :client:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg \
145157
-Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }} \
146158
-Ptarget=${{ matrix.host.TARGET }}
159+
160+
# Then build jedis-compatibility
161+
./gradlew --build-cache :jedis-compatibility:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg \
162+
-Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }} \
163+
-Ptarget=${{ matrix.host.TARGET }}
147164
else
165+
# Build and publish client first
148166
./gradlew --build-cache :client:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg \
149167
-Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }}
168+
169+
# Then build jedis-compatibility
170+
./gradlew --build-cache :jedis-compatibility:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg \
171+
-Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }}
150172
fi
151173
env:
152174
GLIDE_RELEASE_VERSION: ${{ env.RELEASE_VERSION }}
153175

154176
- name: Bundle JAR
155177
working-directory: java
156178
run: |
179+
# Bundle client JAR
157180
src_folder=~/.m2/repository/io/valkey/valkey-glide/${{ env.RELEASE_VERSION }}
158181
cd $src_folder
159182
jar -cvf bundle.jar *
160183
ls -ltr
161184
cd -
162185
cp $src_folder/bundle.jar bundle-${{ matrix.host.TARGET }}.jar
163186
187+
# Bundle jedis-compatibility JAR
188+
jedis_src_folder=~/.m2/repository/io/valkey/valkey-glide-jedis-compatibility/${{ env.RELEASE_VERSION }}
189+
cd $jedis_src_folder
190+
jar -cvf jedis-bundle.jar *
191+
ls -ltr
192+
cd -
193+
cp $jedis_src_folder/jedis-bundle.jar jedis-bundle-${{ matrix.host.TARGET }}.jar
194+
164195
- name: Upload artifacts to publish
165196
continue-on-error: true
166197
uses: actions/upload-artifact@v4
167198
with:
168199
name: java-${{ matrix.host.TARGET }}
169200
path: |
170201
java/bundle*.jar
202+
java/jedis-bundle*.jar
171203
172204
publish-to-maven-central-deployment:
173205
if: ${{ inputs.maven_publish == true || github.event_name == 'push' }}
@@ -196,8 +228,16 @@ jobs:
196228
- name: Move files to the correct directory tree
197229
run: |
198230
mkdir -p build/io/valkey/valkey-glide/${{ env.RELEASE_VERSION }}
199-
cp -a maven-files/* build/io/valkey/valkey-glide/${{ env.RELEASE_VERSION }}
231+
mkdir -p build/io/valkey/valkey-glide-jedis-compatibility/${{ env.RELEASE_VERSION }}
232+
233+
# Move jedis-compatibility files first
234+
cp -a maven-files/valkey-glide-jedis-compatibility* build/io/valkey/valkey-glide-jedis-compatibility/${{ env.RELEASE_VERSION }}
235+
236+
# Move client files (exclude jedis-compatibility files)
237+
find maven-files -name "valkey-glide-*" ! -name "*jedis-compatibility*" -exec cp {} build/io/valkey/valkey-glide/${{ env.RELEASE_VERSION }}/ \;
238+
200239
rm -rf build/io/valkey/valkey-glide/${{ env.RELEASE_VERSION }}/META-INF
240+
rm -rf build/io/valkey/valkey-glide-jedis-compatibility/${{ env.RELEASE_VERSION }}/META-INF
201241
cd build
202242
zip -r ../build io
203243
@@ -253,6 +293,9 @@ jobs:
253293
fail-fast: false
254294
matrix:
255295
host: ${{ fromJson(needs.load-platform-matrix.outputs.PLATFORM_MATRIX) }}
296+
exclude:
297+
- host:
298+
TARGET: aarch64-unknown-linux-musl
256299
runs-on: ${{ matrix.host.test-runner || matrix.host.runner }}
257300
container:
258301
image: ${{ matrix.host.IMAGE || ''}}

.github/workflows/java.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -331,8 +331,13 @@ jobs:
331331
- name: Setup Rust Build
332332
if: ${{ contains(matrix.host.TARGET, 'musl') }}
333333
run: |
334+
export PATH="$HOME/.cargo/bin:$PATH"
334335
echo "PATH=$HOME/.cargo/bin:$PATH" >> $GITHUB_ENV
335336
337+
# Install ziglang and zigbuild
338+
pip3 install ziglang --break-system-packages
339+
cargo install --locked cargo-zigbuild
340+
336341
- uses: actions/cache@v4
337342
with:
338343
path: |

.github/workflows/npm-cd.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -447,9 +447,9 @@ jobs:
447447
uses: geekyeggo/delete-artifact@v2
448448
with:
449449
name: |
450-
npm-packages-${{ github.run_id }}
451450
artifacts-*
452451
js-files-${{ github.run_id }}
452+
failOnError: false
453453

454454
test-published-release:
455455
needs: [get-build-parameters, prepare-and-publish]
@@ -550,9 +550,9 @@ jobs:
550550
uses: geekyeggo/delete-artifact@v2
551551
with:
552552
name: |
553-
npm-packages-${{ github.run_id }}
554553
bindings-*
555554
js-files-${{ github.run_id }}
555+
failOnError: false
556556

557557
deprecate-on-failure:
558558
needs: [test-published-release, get-build-parameters]

CONTRIBUTING.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ Looking at the existing issues is a great way to find something to contribute on
4141
- [Java](./java/DEVELOPER.md)
4242
- [Node](./node/DEVELOPER.md)
4343
- [Python](./python/DEVELOPER.md)
44+
- [Go](./go/DEVELOPER.md)
4445

4546
## Code of Conduct
4647
This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).

go/DEVELOPER.md

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -141,15 +141,7 @@ Before starting this step, make sure you've installed all software requirements.
141141
make build
142142
```
143143

144-
4. Run tests:
145-
1. Ensure that you have installed valkey-server and valkey-cli on your host. You can find the Valkey installation guide at the following link: [Valkey Installation Guide](https://valkey.io/topics/installation/).
146-
2. Execute the following command from the go folder:
147-
148-
```bash
149-
go test -race ./...
150-
```
151-
152-
5. Install Go development tools with:
144+
4. Install Go development tools with:
153145

154146
```bash
155147
make install-dev-tools

go/Makefile

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,27 +10,39 @@ TARGET_DIR := $(GLIDE_FFI_PATH)/target
1010
# Determine the target folder based on OS and architecture
1111
UNAME := $(shell uname)
1212
ARCH := $(shell uname -m)
13+
IS_ALPINE := $(shell test -f /etc/alpine-release && echo true || echo false)
1314

1415
# Set default values for GLIDE_NAME and GLIDE_VERSION
1516
# GLIDE_VERSION is automatically set during the deployment workflow based on the value defined in go-cd.yml.
1617
# For local builds, you can manually specify the version using `GLIDE_VERSION=<version> make build`
1718
GLIDE_NAME = GlideGo
1819
GLIDE_VERSION ?= unknown
1920

21+
ifndef TARGET_TRIPLET
2022
ifeq ($(UNAME), Darwin)
2123
TARGET_TRIPLET := $(if $(filter arm64,$(ARCH)),aarch64-apple-darwin,x86_64-apple-darwin)
24+
else ifeq ($(UNAME), Linux)
25+
ifeq ($(IS_ALPINE), true)
26+
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-musl,x86_64-unknown-linux-musl)
27+
else
28+
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-gnu,x86_64-unknown-linux-gnu)
29+
endif
30+
else
31+
$(error Unsupported platform: $(UNAME) $(ARCH))
32+
endif
33+
endif
34+
35+
ifeq ($(UNAME), Darwin)
2236
# https://github.com/rust-lang/rust/issues/51009#issuecomment-2274649980
2337
BUILD_CMD := rustc --crate-type staticlib
2438
CARGO_FIX_CMD := : # no-op
2539
CARGO_POSTFIX_CMD := : # no-op
2640
STRIP_CMD := strip -x
2741
else ifeq ($(UNAME), Linux)
28-
# TODO: musl
29-
TARGET_TRIPLET := $(if $(filter arm64 aarch64,$(ARCH)),aarch64-unknown-linux-gnu,x86_64-unknown-linux-gnu)
3042
# zigbuild https://github.com/rust-cross/cargo-zigbuild
31-
BUILD_CMD := zigbuild --target $(TARGET_TRIPLET).2.17
43+
BUILD_CMD := zigbuild --target $(TARGET_TRIPLET)
3244
# workaround for https://github.com/rust-cross/cargo-zigbuild/issues/337
33-
CARGO_FIX_CMD := sed -i 's/crate-type.*/crate-type = ["staticlib"]/g' Cargo.toml
45+
CARGO_FIX_CMD := sed -i 's/crate-type.*/crate-type = ["staticlib"]/g' Cargo.toml && rustup target add $(TARGET_TRIPLET)
3446
CARGO_POSTFIX_CMD := git restore Cargo.toml
3547
STRIP_CMD := strip --strip-unneeded
3648
TARGET_DIR := $(TARGET_DIR)/$(TARGET_TRIPLET)
@@ -41,6 +53,15 @@ endif
4153
# Path where compiled binary is copied to and therefore used by go
4254
DEST_PATH := rustbin/$(TARGET_TRIPLET)
4355

56+
# Determine if we're using musl
57+
IS_MUSL := $(if $(filter true,$(IS_ALPINE)),true,$(if $(findstring musl,$(TARGET_TRIPLET)),true,false))
58+
59+
# Set GOFLAGS for musl to use musl tag
60+
ifeq ($(IS_MUSL), true)
61+
export GOFLAGS := -tags=musl
62+
endif
63+
64+
4465
install-build-tools:
4566
go install google.golang.org/protobuf/cmd/[email protected]
4667
cargo install cbindgen
@@ -171,10 +192,21 @@ long-timeout-test: __it
171192
opentelemetry-test: export TEST_FILTER = -skip "TestGlideTestSuite/TestModule" -otel-test $(if $(test-filter), -testify.m $(test-filter), -testify.m TestOpenTelemetry)
172193
opentelemetry-test: __it
173194

195+
# Test command for integration tests
196+
ifeq ($(IS_MUSL), true)
197+
# musl gcc does not support -fsanitize=address, so we use clang instead
198+
# TODO - Investigate musl + address sanitizer issue - https://github.com/valkey-io/valkey-glide/issues/4671
199+
# Disabled for now
200+
# MUSL_TEST_CMD := CC=clang go test -asan -v ./integTest/...
201+
TEST_CMD := CC="gcc" go test -v ./integTest/...
202+
else
203+
TEST_CMD := CC="gcc -fsanitize=address" go test -v ./integTest/...
204+
endif
205+
174206
__it:
175207
mkdir -p reports
176208
set -o pipefail; \
177-
CC="gcc -fsanitize=address" go test -v ./integTest/... \
209+
$(TEST_CMD) \
178210
$(TEST_FILTER) \
179211
$(if $(filter true, $(tls)), --tls,) \
180212
$(if $(standalone-endpoints), --standalone-endpoints=$(standalone-endpoints)) \

0 commit comments

Comments
 (0)