Vault on Kubernetes

Install Vault on K8S with integrated storage via Helm

Posted by Jamie on Saturday, March 25, 2023

What is Vault?

Vault introduction

HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly
control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated by authentication and authorization methods.Using Vault’s UI, CLI, or HTTP API, access to secrets and other sensitive data can be securely stored and managed, tightly controlled (restricted), and auditable.

A modern system requires access to a multitude of secrets, including database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. It can be difficult to understand who is accessing which secrets, especially since this can be platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.

Key features of Vault

  • Secure Secret Storage - Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets.
  • Dynamic Secrets - Vault can generate secrets on-demand for some systems.
  • Data Encryption - Vault can encrypt and decrypt data without storing it.
  • Leasing and Renewal - All secrets in Vault have a lease associated with them, At the end of the lease, Vault will automatically revoke that secret.
  • Revocation - Vault has built-in support for secret revocation. Vault can revoke not only single secrets, but a tree of secrets.

Why do we deploy Vault on K8s?

Kubernetes has become the industry standard for container orchestration solutions, while for application level secrets
management, the kubernetes Secrets are Base64 encoded strings, so there is very high risk of data leakage, it’s recommended to Use an external secret store provider to ensure these secrets are stored and retrieved as securely as possible. And on bear metal machines, to deploy Vault On Kubernetes is a good alternative.

Steps to deploy Vault on K8s

Generate Vault TLS certs

Create private key

  1. Export working directory and naming variables.
export VAULT_K8S_NAMESPACE="vault" \
export VAULT_HELM_RELEASE_NAME="vault" \
export VAULT_SERVICE_NAME="vault-internal" \
export K8S_CLUSTER_NAME="cluster.local" \
export WORKDIR=$(pwd)

root@master:/opt/k8s/config-repo/vault/pki# echo $WORKDIR
/opt/k8s/config-repo/vault/pki

  1. Generate the private key
openssl genrsa -out ${WORKDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus
....................+++
....+++

Create the Certificate Signing Request (CSR).

  1. Create the CSR configuration file
cat > ${WORKDIR}/vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_SERVICE_NAME}
DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
DNS.3 = *.${VAULT_K8S_NAMESPACE}
DNS.4 = *.it-meta.space
DNS.5 = *.vault.svc
IP.1 = 127.0.0.1
IP.2 = 192.168.179.128
EOF

ps:
DNS.4 = *.it-meta.space –> This is for ingress tls proxy.
DNS.5 = *.vault.svc –> It seems required for vault inject agent.
IP.1 = 127.0.0.1 –> Local https protocol inside pod of vault.
IP.2 = 192.168.179.128 –> It’s optional, just for connectivity test using curl command.

  1. Generate the CSR
openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf

Issue the Certificate

  1. Create the csr yaml file to send it to Kubernetes.
cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
   name: vault.svc
spec:
   signerName: kubernetes.io/kubelet-serving
   #expirationSeconds: 8640000
   expirationSeconds: 31536000
   request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')
   usages:
   - digital signature
   - key encipherment
   - server auth
EOF
  1. Send the CSR to Kubernetes.
kubectl create -f ${WORKDIR}/csr.yaml
  1. Approve the CSR in Kubernetes.
kubectl certificate approve vault.svc
  1. Confirm the certificate was issued.
$ kubectl get csr vault.svc
NAME        AGE   SIGNERNAME                      REQUESTOR          REQUESTEDDURATION   CONDITION
vault.svc   75s   kubernetes.io/kubelet-serving   kubernetes-admin   365d                Approved,Issued
```shell
### Store the certificates and Key in the Kubernetes secrets store
1. Retrieve the certificate.
```shell
kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt
  1. Retrieve Kubernetes CA certificate.
kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > ${WORKDIR}/vault.ca
  1. Create the Kubernetes namespace.
kubectl create namespace $VAULT_K8S_NAMESPACE
  1. Create the TLS secret.
kubectl create secret generic vault-ha-tls \
   -n $VAULT_K8S_NAMESPACE \
   --from-file=vault.key=${WORKDIR}/vault.key \
   --from-file=vault.crt=${WORKDIR}/vault.crt \
   --from-file=vault.ca=${WORKDIR}/vault.ca

Deploy the vault cluster via helm with overrides

Prerequisites

Please ensure the Vault CLI, Helm CLI are installed before proceed consequent steps.

root@master:/opt/k8s/config-repo/vault/pki# helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}
root@master:/opt/k8s/config-repo/vault/pki# vault -v
Vault v1.13.0 (a4cf0dc4437de35fce4860857b64569d092a9b5a), built 2023-03-01T14:58:13Z

Deploy HA cluster via Helm

  1. create the overrides.yaml file.
cat > ${WORKDIR}/overrides.yaml <<EOF
global:
  enabled: true
  tlsDisable: false
injector:
  enabled: true
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
server:
  resources:
    requests:
      memory: 256Mi
      cpu: 200m
    limits:
      memory: 384Mi
      cpu: 200m
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
  volumes:
    - name: userconfig-vault-ha-tls
      secret:
        defaultMode: 420
        secretName: vault-ha-tls
  volumeMounts:
    - mountPath: /vault/userconfig/vault-ha-tls
      name: userconfig-vault-ha-tls
      readOnly: true
  auditStorage:
    enabled: true
  standalone:
    enabled: false
  standalone:
    enabled: false
  affinity: ""
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
           tls_disable = 0
           address = "[::]:8200"
           cluster_address = "[::]:8201"
           tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
           tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
           tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
        }
        storage "raft" {
           path = "/vault/data"
        }
        disable_mlock = true
        service_registration "kubernetes" {}        
EOF
  1. Deploy the Vault HA Cluster
helm install -n $VAULT_K8S_NAMESPACE $VAULT_HELM_RELEASE_NAME hashicorp/vault -f ${WORKDIR}/overrides.yaml
  1. Display the pods in the namespace that you created for vault.
root@master:/opt/k8s/config-repo/vault# kubectl -n $VAULT_K8S_NAMESPACE get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          28s
vault-1                                 0/1     Running   0          27s
vault-agent-injector-57bfcf947c-2p6ws   1/1     Running   0          38s
  1. Initialize vault-0 with one key share and one key threshold.
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \
    -key-shares=5 \
    -key-threshold=2 \
    -format=json > ${WORKDIR}/cluster-keys.json

PS:

  1. The operator init command generates a root key that it disassembles into key shares -key-shares=5 and then sets the number of key shares required to unseal Vault -key-threshold=2.
  2. For production HA, better to set the key-shares & key-threshold bit larger than 1 to fit for your security needs. I will set the key-shares as 5 and key-threshold=2 for later demo.
  1. Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key
    randomly pick up two from these 5 shares.
export VAULT_UNSEAL_KEY_1="+BLXCXRatpd5uiamYAly5vfea1eRBOOP8jrRpTxEOzTT"
export VAULT_UNSEAL_KEY_2="K8lYi9kLRA9nQ2KrV0qWvKMjvIREDroJOP7q2OWscljL" 
  1. Unseal Vault running on the vault-0 pod.
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY_1
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY_2

Join vault-1 pod to the Raft cluster

  1. Start an interactive shell session on the vault-1 pod.
kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-1 -- /bin/sh
  1. Join the vault-1 pod to the Raft cluster.
vault operator raft join -address=https://vault-1.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200

Example output:

Key       Value
---       -----
Joined    true

Once success, then exit the vault-1 pod. 3. Unseal vault-1.

kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY_1
kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY_2

Example output:

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          2
Unseal Progress    0/2
Unseal Nonce       n/a
Version            1.12.1
Build Date         2022-10-27T12:32:05Z
Storage Type       raft
HA Enabled         true
  1. Export the cluster root token.
export CLUSTER_ROOT_TOKEN=$(cat ${WORKDIR}/cluster-keys.json | jq -r ".root_token")
  1. Login to vault-0 with the root token.
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault login $CLUSTER_ROOT_TOKEN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.oemGqKix23q5FMi6qkXiLsqN
token_accessor       BVDmPTEBLgdxLEvQs4p5bXaD
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
  1. List the raft peers.
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator raft list-peers
Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
  1. Print the HA status
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               2
Version                 1.12.1
Build Date              2022-10-27T12:32:05Z
Storage Type            raft
Cluster Name            vault-cluster-1ff6c3b7
Cluster ID              989f282b-a491-6acb-7ad2-24401d96ff0c
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2023-03-26T05:15:13.133233643Z
Raft Committed Index    40
Raft Applied Index      40

We now have a working 2 node cluster with TLS enabled at the pod level. Next we will create a secret and retrieve it via and API call to confirm TLS is working as expected.

Create ingress to expose the Vault UI(not recommend on Production)

  1. Prepare TLS - request free cert on AliCloud. Here i got 1 free cert pair for my domain vault.it-meta.space
root@master:/opt/k8s/config-repo/vault# ls -al | grep it-meta.space
-rwxrwxrwx 1 jamie jamie 1679 Mar 14 14:54 9482518_vault.it-meta.space.key
-rwxrwxrwx 1 jamie jamie 3813 Mar 14 14:54 9482518_vault.it-meta.space.pem
  1. Create Secret tls from these two cert files
k create secret tls vault-ext-tls --cert=./9482518_vault.it-meta.space.pem --key=./9482518_vault.it-meta.space.key -n vault
  1. Create ingress for vault UI
cat > ${WORKDIR}/vault-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault-ingress
  namespace: vault
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
    - host: vault.it-meta.space
      http:
        paths:
          - path: /(.*)
            pathType: Prefix
            backend:
              service:
                name: vault
                port:
                  number: 8200
  tls:
    - secretName: vault-ext-tls
      hosts:
        - vault.it-meta.space
EOF

k apply -f ${WORKDIR}/vault-ingress.yaml
  1. Verify the ingress.

Use root token to login

Continue

Ends here, will continue the kubernetes engine enablement and vault inject agent in next blog.

Reference docs:
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls

「真诚赞赏,手留余香」

Jamie's Blog

真诚赞赏,手留余香

使用微信扫描二维码完成支付