Skip to content

Commit 38b41ac

Browse files
Shobhit-IBMShobhit-IBM
committed
vault 3 node cluster deployment on zcx environment
1 parent e46c5e2 commit 38b41ac

File tree

3 files changed

+268
-0
lines changed

3 files changed

+268
-0
lines changed

‎content/vault/v1.21.x/content/docs/deploy/run-as-zcx-cluster.mdx‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
---
2+
layout: docs
3+
page_title: Run Vault as zcx cluster
4+
description: >-
5+
Step-by-step guide to deploying a secure 3-node Vault cluster inside zCX using HAProxy and TLS.
6+
---
7+
8+
# Run Vault as zcx cluster
9+
10+
This guide walks through deploying a fully secured 3-node HashiCorp Vault Enterprise cluster
11+
on IBM z/OS Container Extensions (zCX). The setup uses three independent zCX instances,
12+
each running Vault with its own unique IP address. A Layer-4 HAProxy load balancer sits
13+
in front to distribute traffic, and end-to-end TLS encryption is applied across all Vault nodes
14+
and the load balancer to ensure secure communication throughout the cluster.
15+
16+
![Vault zcx Cluster Deployment](/img/run-as-zcx-cluster.png)
17+
18+
## Step 1: Container Image Deployment
19+
Pull the official Vault Enterprise container image on all three zCX nodes.
20+
```shell-session
21+
$ docker pull hashicorp/vault-enterprise:1.19.12-ent
22+
```
23+
Verify image integrity
24+
```shell-session
25+
$ docker images | grep vault-enterprise
26+
```
27+
28+
## Step 2: Persistent Volume Creation
29+
Create Docker volumes for configuration and data persistence on each node.
30+
31+
Create volume for Vault data storage (Raft backend, audit logs)
32+
```shell-session
33+
$ docker volume create vault-data
34+
```
35+
Create volume for Vault configuration files
36+
```shell-session
37+
$ docker volume create vault-config
38+
```
39+
Verify volume creation
40+
```shell-session
41+
$ docker volume ls | grep vault
42+
```
43+
44+
Volume Mount Points:
45+
- vault-data: /vault/data (stores Raft snapshots, encrypted storage backend)
46+
- vault-config: /vault/config.d (HCL configuration files)
47+
48+
## Step 4: Vault Configuration File Setup
49+
Create Vault configuration on each node with node-specific parameters:
50+
51+
Access config volume
52+
```shell-session
53+
$ docker run --rm -it -v vault-config:/vault alpine sh
54+
```
55+
56+
Create configuration file for Leader and follower vault
57+
```hcl
58+
ui = true
59+
disable_mlock = true
60+
61+
# --- Listener Configuration ---
62+
listener "tcp" {
63+
address = "0.0.0.0:8200"
64+
tls_disable = 0
65+
tls_cert_file = "/vault/vault.pem"
66+
tls_key_file = "/vault/vault.key"
67+
tls_client_ca_file = "/vault/ca.pem"
68+
}
69+
70+
# --- Raft Storage (Leader Node) ---
71+
storage "raft" {
72+
path = "/vault/data"
73+
node_id = "<LEADER_NODE_ID>" # e.g., "node1"
74+
75+
# No retry_join block needed for leader
76+
}
77+
78+
# --- Cluster Networking ---
79+
api_addr = "https://<LEADER_IP>:8200"
80+
cluster_addr = "https://<LEADER_IP>:8201"
81+
```
82+
Exposes Vault on 8200 for API and UI traffic.
83+
84+
- tls_disable = 0 → TLS is enabled (mandatory for secure cluster communication).
85+
86+
### Certificates:
87+
88+
- vault.pem → TLS certificate
89+
90+
- vault.key → Private key
91+
92+
- ca.pem → CA used to validate mutual TLS between nodes
93+
94+
This ensures encrypted traffic between Vault clients, HAProxy, and other nodes.
95+
96+
Similarly create configuration for Follower Vault Nodes
97+
98+
```hcl
99+
ui = true
100+
disable_mlock = true
101+
102+
# --- Listener Configuration ---
103+
listener "tcp" {
104+
address = "0.0.0.0:8200"
105+
tls_disable = 0
106+
tls_cert_file = "/vault/vault.pem"
107+
tls_key_file = "/vault/vault.key"
108+
tls_client_ca_file = "/vault/ca.pem"
109+
}
110+
111+
# --- Raft Storage (Follower Node) ---
112+
storage "raft" {
113+
path = "/vault/data"
114+
node_id = "<FOLLOWER_NODE_ID>" # e.g., "node2" or "node3"
115+
116+
retry_join {
117+
leader_api_addr = "https://<LEADER_IP>:8200"
118+
leader_ca_cert_file = "/vault/ca.pem"
119+
}
120+
}
121+
122+
# --- Cluster Networking ---
123+
api_addr = "https://<FOLLOWER_IP>:8200"
124+
cluster_addr = "https://<FOLLOWER_IP>:8201"
125+
```
126+
127+
## Step 5: Vault Container Deployment
128+
Deploy Vault container on each zCX node.
129+
130+
Set Vault Enterprise license (export before running)
131+
```plaintext
132+
export VAULT_LICENSE="02MV4UU43BK5HGYYTOJZWFQMTMNNEWU33JJVVGC..." # Replace with actual license
133+
```
134+
135+
Deploying the Vault Enterprise Container on zCX
136+
```shell-session
137+
$ docker run -d \
138+
--name vault-zcx \
139+
--cap-add IPC_LOCK \
140+
-p 8200:8200 \
141+
-p 8201:8201 \
142+
-v vault-config:/vault/config.d \
143+
-v vault-data:/vault/data \
144+
-e VAULT_LICENSE="$VAULT_LICENSE" \
145+
hashicorp/vault-enterprise:1.19.12-ent \
146+
server -config=/vault/config.d/
147+
```
148+
Verify container status
149+
```shell-session
150+
docker ps | grep vault-zcx
151+
docker logs vault-zcx
152+
```
153+
154+
## Cluster Initialization and Unsealing
155+
Initialize Vault Cluster (Run on ONE node only)
156+
157+
Initialize the Leader Vault
158+
```shell-session
159+
docker exec -it vault-zcx vault operator init \
160+
-format=json > /secure/location/vault-init.json
161+
```
162+
Extract unseal keys and root token
163+
```shell-session
164+
cat /secure/location/vault-init.json
165+
```
166+
Unseal Vault on All Nodes
167+
168+
Unseal with 3 different keys (repeat on each node)
169+
```shell-session
170+
docker exec -it vault-zcx vault operator unseal <UNSEAL_KEY_1>
171+
docker exec -it vault-zcx vault operator unseal <UNSEAL_KEY_2>
172+
docker exec -it vault-zcx vault operator unseal <UNSEAL_KEY_3>
173+
```
174+
Remember to unseal with same key that you got from leader
175+
Check seal status
176+
```shell-session
177+
vault status
178+
```
179+
180+
## HAProxy Load Balancer
181+
182+
In a Vault cluster, the load balancer plays a critical role in directing traffic to the active leader
183+
while maintaining full security. In this example, we are using an HAProxy Layer-4 (TCP) load balancer,
184+
but you can use other load balancers as well depending on your environment and requirements.
185+
186+
- End-to-end TLS encryption: L4 simply forwards TCP connections, so Vault’s TLS traffic remains fully encrypted without terminating SSL at the load balancer.
187+
188+
- Simpler configuration: No need to manage certificates on HAProxy or re-encrypt traffic.
189+
190+
- Leader-aware routing: Vault handles leader redirects natively, so the load balancer doesn’t need to interpret HTTP or Vault protocols.
191+
192+
- Better performance: Forwarding raw TCP reduces overhead, giving lower latency and higher throughput.
193+
194+
### Configuration
195+
```plaintext
196+
global
197+
maxconn 4096
198+
log stdout format raw local0
199+
200+
defaults
201+
mode tcp # TCP mode for Layer-4 forwarding
202+
timeout connect 5s
203+
timeout client 1m
204+
timeout server 1m
205+
log global
206+
207+
# --- Stats Page (HTTPS) ---
208+
frontend stats
209+
bind *:8404 ssl crt /usr/local/etc/haproxy/haproxy.pem
210+
mode http
211+
stats enable
212+
stats uri /stats
213+
stats refresh 10s
214+
stats admin if TRUE
215+
216+
# --- Vault API Frontend (TLS Passthrough) ---
217+
frontend vault_api
218+
bind *:8200 # Listen on Vault API port
219+
mode tcp # Layer-4 forwarding to preserve TLS
220+
default_backend vault_nodes
221+
222+
# --- Vault Nodes Backend ---
223+
backend vault_nodes
224+
mode tcp
225+
balance roundrobin # Simple load distribution among nodes
226+
option tcp-check # Health check at TCP level
227+
server node1 <NODE1_IP>:8200 check
228+
server node2 <NODE2_IP>:8200 check
229+
server node3 <NODE3_IP>:8200 check
230+
```
231+
Copy config to volume
232+
```shell-session
233+
docker cp haproxy.cfg haproxy-tmp:/usr/local/etc/haproxy/haproxy.cfg
234+
```
235+
Deploy haproxy load-balancer
236+
```shell-session
237+
docker run -d \
238+
--name vault-lb \
239+
-p 8300:8200 \
240+
-p 8404:8404 \
241+
-v haproxy-config:/usr/local/etc/haproxy \
242+
ibmz-hc-registry.ngrok.dev/haproxy:3.2
243+
```
244+
245+
## Verify Cluster
246+
List all the leader and follower nodes
247+
```shell-session
248+
vault operator raft list-peers
249+
```
250+
251+
Test container health on browser using haproxy endpoint
252+
```plaintext
253+
https://<LOAD_BALANCER_IP>:<PORT>/stats
254+
```
255+
256+
257+
Test Secure Vault Connection and View Raft Configuration
258+
```shell-session
259+
curl \
260+
--cacert <CA_CERT_FILE> \
261+
--header "X-Vault-Token: <VAULT_TOKEN>" \
262+
https://<LOAD_BALANCER_IP>:<PORT>/v1/sys/storage/raft/configuration \
263+
| jq .
264+
```

‎content/vault/v1.21.x/data/docs-nav-data.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -724,6 +724,10 @@
724724
"title": "Run as a service",
725725
"path": "deploy/run-as-service"
726726
},
727+
{
728+
"title": "Run as a zcx cluster",
729+
"path": "deploy/run-as-zcx-cluster"
730+
},
727731
{
728732
"title": "Run on AWS",
729733
"routes": [

‎content/vault/v1.21.x/img/run-as-zcx-cluster.png‎

75.8 KB
Loading

0 commit comments

Comments
 (0)