What is Vault?
Vault introduction
HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly
control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated
by authentication and authorization methods.Using Vault’s UI, CLI, or HTTP API, access to secrets and other sensitive data
can be securely stored and managed, tightly controlled (restricted), and auditable.
A modern system requires access to a multitude of secrets, including database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. It can be difficult to understand who is accessing which secrets, especially since this can be platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.
Key features of Vault
- Secure Secret Storage - Arbitrary key/value secrets can be stored in Vault. Vault encrypts these secrets prior to writing them to persistent storage, so gaining access to the raw storage isn’t enough to access your secrets.
- Dynamic Secrets - Vault can generate secrets on-demand for some systems.
- Data Encryption - Vault can encrypt and decrypt data without storing it.
- Leasing and Renewal - All secrets in Vault have a lease associated with them, At the end of the lease, Vault will automatically revoke that secret.
- Revocation - Vault has built-in support for secret revocation. Vault can revoke not only single secrets, but a tree of secrets.
Why do we deploy Vault on K8s?
Kubernetes has become the industry standard for container orchestration solutions, while for application level secrets
management, the kubernetes Secrets are Base64 encoded strings, so there is very high risk of data leakage, it’s
recommended to Use an external secret store provider to ensure these secrets are stored and retrieved as securely
as possible. And on bear metal machines, to deploy Vault On Kubernetes is a good alternative.
Steps to deploy Vault on K8s
Generate Vault TLS certs
Create private key
- Export working directory and naming variables.
export VAULT_K8S_NAMESPACE="vault" \
export VAULT_HELM_RELEASE_NAME="vault" \
export VAULT_SERVICE_NAME="vault-internal" \
export K8S_CLUSTER_NAME="cluster.local" \
export WORKDIR=$(pwd)
root@master:/opt/k8s/config-repo/vault/pki# echo $WORKDIR
/opt/k8s/config-repo/vault/pki
- Generate the private key
openssl genrsa -out ${WORKDIR}/vault.key 2048
Generating RSA private key, 2048 bit long modulus
....................+++
....+++
Create the Certificate Signing Request (CSR).
- Create the CSR configuration file
cat > ${WORKDIR}/vault-csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
encrypt_key = yes
default_md = sha256
distinguished_name = kubelet_serving
req_extensions = v3_req
[ kubelet_serving ]
O = system:nodes
CN = system:node:*.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${VAULT_SERVICE_NAME}
DNS.2 = *.${VAULT_SERVICE_NAME}.${VAULT_K8S_NAMESPACE}.svc.${K8S_CLUSTER_NAME}
DNS.3 = *.${VAULT_K8S_NAMESPACE}
DNS.4 = *.it-meta.space
DNS.5 = *.vault.svc
IP.1 = 127.0.0.1
IP.2 = 192.168.179.128
EOF
ps:
DNS.4 = *.it-meta.space –> This is for ingress tls proxy.
DNS.5 = *.vault.svc –> It seems required for vault inject agent.
IP.1 = 127.0.0.1 –> Local https protocol inside pod of vault.
IP.2 = 192.168.179.128 –> It’s optional, just for connectivity test using curl command.
- Generate the CSR
openssl req -new -key ${WORKDIR}/vault.key -out ${WORKDIR}/vault.csr -config ${WORKDIR}/vault-csr.conf
Issue the Certificate
- Create the csr yaml file to send it to Kubernetes.
cat > ${WORKDIR}/csr.yaml <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: vault.svc
spec:
signerName: kubernetes.io/kubelet-serving
#expirationSeconds: 8640000
expirationSeconds: 31536000
request: $(cat ${WORKDIR}/vault.csr|base64|tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
- Send the CSR to Kubernetes.
kubectl create -f ${WORKDIR}/csr.yaml
- Approve the CSR in Kubernetes.
kubectl certificate approve vault.svc
- Confirm the certificate was issued.
$ kubectl get csr vault.svc
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
vault.svc 75s kubernetes.io/kubelet-serving kubernetes-admin 365d Approved,Issued
```shell
### Store the certificates and Key in the Kubernetes secrets store
1. Retrieve the certificate.
```shell
kubectl get csr vault.svc -o jsonpath='{.status.certificate}' | openssl base64 -d -A -out ${WORKDIR}/vault.crt
- Retrieve Kubernetes CA certificate.
kubectl config view \
--raw \
--minify \
--flatten \
-o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
| base64 -d > ${WORKDIR}/vault.ca
- Create the Kubernetes namespace.
kubectl create namespace $VAULT_K8S_NAMESPACE
- Create the TLS secret.
kubectl create secret generic vault-ha-tls \
-n $VAULT_K8S_NAMESPACE \
--from-file=vault.key=${WORKDIR}/vault.key \
--from-file=vault.crt=${WORKDIR}/vault.crt \
--from-file=vault.ca=${WORKDIR}/vault.ca
Deploy the vault cluster via helm with overrides
Prerequisites
Please ensure the Vault CLI, Helm CLI are installed before proceed consequent steps.
root@master:/opt/k8s/config-repo/vault/pki# helm version
version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}
root@master:/opt/k8s/config-repo/vault/pki# vault -v
Vault v1.13.0 (a4cf0dc4437de35fce4860857b64569d092a9b5a), built 2023-03-01T14:58:13Z
Deploy HA cluster via Helm
- create the overrides.yaml file.
cat > ${WORKDIR}/overrides.yaml <<EOF
global:
enabled: true
tlsDisable: false
injector:
enabled: true
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m
server:
resources:
requests:
memory: 256Mi
cpu: 200m
limits:
memory: 384Mi
cpu: 200m
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
volumes:
- name: userconfig-vault-ha-tls
secret:
defaultMode: 420
secretName: vault-ha-tls
volumeMounts:
- mountPath: /vault/userconfig/vault-ha-tls
name: userconfig-vault-ha-tls
readOnly: true
auditStorage:
enabled: true
standalone:
enabled: false
standalone:
enabled: false
affinity: ""
ha:
enabled: true
replicas: 2
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
}
storage "raft" {
path = "/vault/data"
}
disable_mlock = true
service_registration "kubernetes" {}
EOF
- Deploy the Vault HA Cluster
helm install -n $VAULT_K8S_NAMESPACE $VAULT_HELM_RELEASE_NAME hashicorp/vault -f ${WORKDIR}/overrides.yaml
- Display the pods in the namespace that you created for vault.
root@master:/opt/k8s/config-repo/vault# kubectl -n $VAULT_K8S_NAMESPACE get pods
NAME READY STATUS RESTARTS AGE
vault-0 0/1 Running 0 28s
vault-1 0/1 Running 0 27s
vault-agent-injector-57bfcf947c-2p6ws 1/1 Running 0 38s
- Initialize vault-0 with one key share and one key threshold.
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator init \
-key-shares=5 \
-key-threshold=2 \
-format=json > ${WORKDIR}/cluster-keys.json
PS:
- The operator init command generates a root key that it disassembles into key shares -key-shares=5 and then sets the number of key shares required to unseal Vault -key-threshold=2.
- For production HA, better to set the key-shares & key-threshold bit larger than 1 to fit for your security needs. I will set the key-shares as 5 and key-threshold=2 for later demo.
- Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key
randomly pick up two from these 5 shares.
export VAULT_UNSEAL_KEY_1="+BLXCXRatpd5uiamYAly5vfea1eRBOOP8jrRpTxEOzTT"
export VAULT_UNSEAL_KEY_2="K8lYi9kLRA9nQ2KrV0qWvKMjvIREDroJOP7q2OWscljL"
- Unseal Vault running on the vault-0 pod.
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY_1
kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY_2
Join vault-1 pod to the Raft cluster
- Start an interactive shell session on the vault-1 pod.
kubectl exec -n $VAULT_K8S_NAMESPACE -it vault-1 -- /bin/sh
- Join the vault-1 pod to the Raft cluster.
vault operator raft join -address=https://vault-1.vault-internal:8200 -leader-ca-cert="$(cat /vault/userconfig/vault-ha-tls/vault.ca)" -leader-client-cert="$(cat /vault/userconfig/vault-ha-tls/vault.crt)" -leader-client-key="$(cat /vault/userconfig/vault-ha-tls/vault.key)" https://vault-0.vault-internal:8200
Example output:
Key Value
--- -----
Joined true
Once success, then exit the vault-1 pod. 3. Unseal vault-1.
kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY_1
kubectl exec -n $VAULT_K8S_NAMESPACE -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY_2
Example output:
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 2
Unseal Progress 0/2
Unseal Nonce n/a
Version 1.12.1
Build Date 2022-10-27T12:32:05Z
Storage Type raft
HA Enabled true
- Export the cluster root token.
export CLUSTER_ROOT_TOKEN=$(cat ${WORKDIR}/cluster-keys.json | jq -r ".root_token")
- Login to vault-0 with the root token.
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault login $CLUSTER_ROOT_TOKEN
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.oemGqKix23q5FMi6qkXiLsqN
token_accessor BVDmPTEBLgdxLEvQs4p5bXaD
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
- List the raft peers.
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-0 vault-0.vault-internal:8201 leader true
vault-1 vault-1.vault-internal:8201 follower true
- Print the HA status
root@master:/opt/k8s/config-repo/vault/pki# kubectl exec -n $VAULT_K8S_NAMESPACE vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 2
Version 1.12.1
Build Date 2022-10-27T12:32:05Z
Storage Type raft
Cluster Name vault-cluster-1ff6c3b7
Cluster ID 989f282b-a491-6acb-7ad2-24401d96ff0c
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2023-03-26T05:15:13.133233643Z
Raft Committed Index 40
Raft Applied Index 40
We now have a working 2 node cluster with TLS enabled at the pod level. Next we will create a secret and retrieve it via and API call to confirm TLS is working as expected.
Create ingress to expose the Vault UI(not recommend on Production)
- Prepare TLS - request free cert on AliCloud. Here i got 1 free cert pair for my domain vault.it-meta.space
root@master:/opt/k8s/config-repo/vault# ls -al | grep it-meta.space
-rwxrwxrwx 1 jamie jamie 1679 Mar 14 14:54 9482518_vault.it-meta.space.key
-rwxrwxrwx 1 jamie jamie 3813 Mar 14 14:54 9482518_vault.it-meta.space.pem
- Create Secret tls from these two cert files
k create secret tls vault-ext-tls --cert=./9482518_vault.it-meta.space.pem --key=./9482518_vault.it-meta.space.key -n vault
- Create ingress for vault UI
cat > ${WORKDIR}/vault-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: vault-ingress
namespace: vault
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
ingressClassName: nginx
rules:
- host: vault.it-meta.space
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: vault
port:
number: 8200
tls:
- secretName: vault-ext-tls
hosts:
- vault.it-meta.space
EOF
k apply -f ${WORKDIR}/vault-ingress.yaml
- Verify the ingress.
Use root token to login
Continue
Ends here, will continue the kubernetes engine enablement and vault inject agent in next blog.
Reference docs:
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls
「真诚赞赏,手留余香」
真诚赞赏,手留余香
使用微信扫描二维码完成支付