Mastering Zero-Trust Secrets: Integrating HashiCorp Vault with Kubernetes

1. The Challenge: Beyond Base64

In modern cloud-native environments, the "standard" way of handling secrets (Kubernetes Secret objects) is often insufficient. Kubernetes Secrets are merely Base64 encoded, not encrypted at rest by default. For any organization serious about security, we need Zero-Trust: no hardcoded keys in Git, no sensitive data in Helm charts, and centralized audit trails.

This guide documents the implementation of a High-Availability (HA) Vault cluster and its seamless integration with Spring Boot microservices.

2. The Architecture

Our setup utilizes the Vault Agent Sidecar Injection pattern. Instead of the application "asking" Vault for secrets via API code, Vault "pushes" secrets into the pod as local files.

• Vault Server: HA Cluster using Raft (integrated storage).
• Vault Injector: A mutating admission webhook that monitors pod annotations.
• Auth Method: Kubernetes Auth, where Vault verifies the pod's identity via its ServiceAccount JWT.

Phase 1: Clean Deployment (Helm)

To install Vault with High Availability (HA) and the Web UI enabled, we use the official Helm chart.

# 1. Add repo
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# 2. Install HA Cluster with Raft Storage
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set "server.ui.enabled=true" \
  --set "server.ui.serviceType=NodePort" \
  --set "server.ha.enabled=true" \
  --set "server.ha.raft.enabled=true" \
  --set "server.ha.raft.setNodeId=true"

Phase 2: The "Fortress" Initialization

Vault starts in a "Sealed" state. It cannot read its own data until it is unsealed.

Initializing the Leader

# 1. Initialize (Generating the Master Keys)
# In production, use -key-shares=5 -key-threshold=3
kubectl exec -n vault vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json

# 2. Extract keys for convenience
export UNSEAL_KEY=$(jq -r ".unseal_keys_b64[0]" cluster-keys.json)
export ROOT_TOKEN=$(jq -r ".root_token" cluster-keys.json)

# 3. Unseal the leader (vault-0)
kubectl exec -n vault vault-0 -- vault operator unseal $UNSEAL_KEY

Joining HA Nodes (vault-1 and vault-2)

For a true HA cluster, the other pods must join the Raft cluster:

# Repeat for vault-1 and vault-2
kubectl exec -n vault vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec -n vault vault-1 -- vault operator unseal $UNSEAL_KEY

kubectl exec -n vault vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
kubectl exec -n vault vault-2 -- vault operator unseal $UNSEAL_KEY

Phase 3: Core Configuration

Log into the Vault CLI to enable the basic engines required for Kubernetes Integration.

# Enter the Vault container
kubectl exec -it -n vault vault-0 -- /bin/sh

# Login as Admin
vault login $ROOT_TOKEN

# 1. Enable KV-V2 Secrets Engine (The 'internal' path)
vault secrets enable -path=internal kv-v2

# 2. Enable Kubernetes Authentication
vault auth enable kubernetes

# 3. Configure Auth to talk to the K8s API
vault write auth/kubernetes/config \
    kubernetes_host="https://kubernetes.default.svc:443"

Phase 4: Application Onboarding

Follow these steps for every new microservice you deploy to fetch secrets dynamically.

1. Define the Policy (app-policy.hcl)

Create a policy that limits the app to its own folder. Note: KV-V2 requires the data/ prefix in the path.

vault policy write my-app-policy - <<EOF
path "internal/data/my-app/config" {
  capabilities = ["read"]
}
EOF

2. Create the Role

Binds the Kubernetes ServiceAccount to the Policy.

vault write auth/kubernetes/role/my-app-role \
    bound_service_account_names=my-app-sa \
    bound_service_account_namespaces=production \
    policies=my-app-policy \
    ttl=24h

3. Seed the Secrets

Push the actual sensitive data into Vault.

vault kv put internal/my-app/config \
    DB_PASSWORD="super-secret-pass" \
    API_KEY="AIzaSyB..." \
    S3_SECRET="a6DuZ0E..."

Phase 5: Troubleshooting & Best Practices

The "Permission Denied" (403) Checklist

Best Practice: The Backup (Snapshot)

Vault data is encrypted, but you must still back up the encrypted database.

# Take a snapshot of the Raft database
kubectl exec -n vault vault-0 -- vault operator raft snapshot save /share/vault-backup.snap

Best Practice: Maintenance

Conclusion

By moving secrets into Vault, we didn't just improve security; we drastically improved Developer Experience (DX). Developers no longer manage and pass around secret.yaml files. They simply define what they need directly in Vault, and the infrastructure handles the rest securely and transparently.