🕸️ Ada Research Browser

openbao-sealed.md
Home›Research Reports›openbao-sealed.md
← Back

Runbook: OpenBao Sealed

Alert

Severity

Critical -- When OpenBao is sealed, no secrets can be read or written. All ExternalSecret resources will fail to sync, causing application pods that depend on secrets to fail startup or lose access to rotated credentials.

Impact

Investigation Steps

  1. Check OpenBao pod status:
kubectl get pods -n openbao -l app.kubernetes.io/name=openbao
  1. Check the seal status:
kubectl exec -n openbao openbao-0 -- bao status

  1. If the pod is running but sealed, the output will show Sealed: true

  2. Check OpenBao logs for why it became sealed:

kubectl logs -n openbao openbao-0 --tail=200
  1. Check if the pod restarted (which causes auto-seal):
kubectl describe pod -n openbao openbao-0 | grep -A 5 "Last State"
kubectl get events -n openbao --sort-by='.lastTimestamp' | tail -20
  1. Check the HelmRelease status:
flux get helmrelease openbao -n openbao
  1. Check External Secrets Operator for sync failures:
kubectl get externalsecrets -A
kubectl get clustersecretstores
  1. Check if the unseal keys secret still exists:
kubectl get secret openbao-init-keys -n openbao

Resolution

Standard unseal procedure

OpenBao seals itself on pod restart. The unseal keys are stored in the openbao-init-keys secret in the openbao namespace.

  1. Retrieve the unseal keys:
kubectl get secret openbao-init-keys -n openbao -o jsonpath='{.data.unseal-keys}' | base64 -d
  1. Unseal OpenBao (repeat for each key share required -- default threshold is 1 for dev):
kubectl exec -n openbao openbao-0 -- bao operator unseal <UNSEAL_KEY>
  1. Verify OpenBao is unsealed:
kubectl exec -n openbao openbao-0 -- bao status

  1. Confirm the output shows Sealed: false

  2. Verify External Secrets can sync again:

kubectl get clustersecretstores
kubectl get externalsecrets -A

OpenBao pod crash-looping

  1. Check the pod logs for the crash reason:
kubectl logs -n openbao openbao-0 --previous
  1. Common causes:
  2. Storage backend corruption -- check PVC status
  3. Out of memory -- check resource limits in the HelmRelease

  4. Configuration error after upgrade

  5. If storage is the issue, check the PVC:

kubectl get pvc -n openbao
kubectl describe pvc data-openbao-0 -n openbao
  1. If the pod cannot start, try restarting:
kubectl delete pod openbao-0 -n openbao
  1. After the pod restarts, unseal it (see standard unseal procedure above)

Unseal keys secret is missing

If the openbao-init-keys secret has been deleted:

  1. Check if you have a backup of the unseal keys outside the cluster
  2. If unseal keys are permanently lost, OpenBao data is unrecoverable and must be re-initialized:
kubectl exec -n openbao openbao-0 -- bao operator init -key-shares=1 -key-threshold=1 -format=json
  1. Store the new unseal keys and root token:
kubectl create secret generic openbao-init-keys -n openbao \
  --from-literal=unseal-keys="<NEW_UNSEAL_KEY>" \
  --from-literal=root-token="<NEW_ROOT_TOKEN>"
  1. Unseal with the new key
  2. Reconfigure all secret engines, auth methods, and policies
  3. Re-sync all ExternalSecrets

Re-enable External Secrets sync after unseal

  1. After unsealing, check that the ClusterSecretStore reconnects:
kubectl get clustersecretstores -o wide
  1. Force a sync of all ExternalSecrets:
for es in $(kubectl get externalsecrets -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers); do
  ns=$(echo $es | awk '{print $1}')
  name=$(echo $es | awk '{print $2}')
  kubectl annotate externalsecret $name -n $ns force-sync=$(date +%s) --overwrite
done

Prevention

Escalation