On this page

HashiCorp Vault Deep Dive

Why Vault Exists

HashiCorp Vault is a secrets management platform that centralizes secret storage, provides dynamic secrets with automatic expiration, handles encryption as a service, and enforces fine-grained access policies. It exists because organizations need a single source of truth for secrets that works across clouds, environments, and teams.

Before Vault, secrets were scattered across environment variables, config files, encrypted data bags, and proprietary cloud services. Each approach had limitations — no dynamic secrets, no unified audit trail, no cross-cloud support, no encryption as a service. Vault brought all of these under one roof.

Key Differentiators

FeatureVaultCloud KMS/Secrets ManagerConfig Files
Dynamic secretsYes (DB, AWS, PKI)NoNo
Encryption as a serviceTransit engineEncrypt/Decrypt APINo
Multi-cloudYesSingle cloudManual
Lease managementAutomatic expiryManualNo
Identity-based accessAppRole, K8s, OIDCIAMFile permissions
Audit loggingComprehensiveVariesNo

First Principles

Vault's Architecture

graph TB
    subgraph "Client Layer"
        CLI[Vault CLI]
        SDK[Vault SDK]
        API[HTTP API]
        UI[Web UI]
    end

    subgraph "Vault Core"
        AUTH[Auth Methods] --> CORE[Core / Router]
        CORE --> SE[Secrets Engines]
        CORE --> AUDIT[Audit Devices]
        CORE --> SYS[System Backend]
    end

    subgraph "Storage Backend"
        CONSUL[Consul]
        RAFT[Integrated Raft Storage]
        DB_STORE[Database]
    end

    subgraph "Seal/Unseal"
        SEAL[Seal Mechanism<br/>Shamir / Auto-unseal / HSM]
        BARRIER[Encryption Barrier]
    end

    CLI --> CORE
    SDK --> CORE
    API --> CORE
    UI --> CORE

    SE --> BARRIER
    BARRIER --> RAFT

    SEAL --> BARRIER

    style CORE fill:#e3f2fd
    style BARRIER fill:#ffebee
    style SEAL fill:#fff3e0

The Seal/Unseal Mechanism

Vault encrypts all data with a master key. The master key is protected by the seal mechanism:

stateDiagram-v2
    [*] --> Sealed: Vault starts
    Sealed --> Unsealing: Provide unseal keys
    Unsealing --> Unsealed: Threshold met (3 of 5)
    Unsealed --> Sealed: Manual seal or restart
    Unsealed --> Unsealed: Serving requests

    note right of Sealed: All data encrypted,<br/>no operations possible
    note right of Unsealed: Master key in memory,<br/>serving requests

Shamir's Secret Sharing: The master key is split into N shares, requiring K shares to reconstruct:

Typically: 5 shares, threshold of 3. Each share is given to a different operator.

Auto-unseal: For automated environments, use cloud KMS (AWS KMS, GCP CKMS, Azure Key Vault) or HSM to unseal automatically.

Path-Based Access Model

Every secret in Vault has a path, and access is controlled by policies:

secret/                          # Root of KV secrets engine
secret/data/production/          # Production secrets
secret/data/production/db        # Database credentials
secret/data/staging/db           # Staging database credentials
aws/creds/my-role               # Dynamic AWS credentials
database/creds/readonly         # Dynamic database credentials
pki/issue/web-server            # Dynamic TLS certificates

Core Mechanics

Secrets Engines

graph TB
    subgraph "Static Secrets"
        KV[KV v2<br/>Key-Value Store]
    end

    subgraph "Dynamic Secrets"
        AWS_SE[AWS Engine<br/>IAM credentials]
        DB_SE[Database Engine<br/>DB credentials]
        PKI[PKI Engine<br/>X.509 certificates]
        SSH_SE[SSH Engine<br/>SSH certificates]
        CONSUL_SE[Consul Engine<br/>Consul tokens]
    end

    subgraph "Encryption"
        TRANSIT[Transit Engine<br/>Encryption as a Service]
        TRANSFORM[Transform Engine<br/>Format-Preserving Encryption]
    end

    style KV fill:#e8f5e9
    style AWS_SE fill:#e3f2fd
    style TRANSIT fill:#fff3e0

Dynamic Secrets — Database Engine

Dynamic database credentials are generated on-demand and automatically revoked:

sequenceDiagram
    participant APP as Application
    participant V as Vault
    participant DB as PostgreSQL

    APP->>V: GET /database/creds/readonly
    V->>V: Check policy, verify auth
    V->>DB: CREATE ROLE "v-app-readonly-abc123" WITH LOGIN PASSWORD 'random'
    V->>DB: GRANT SELECT ON ALL TABLES TO "v-app-readonly-abc123"
    V->>V: Create lease (TTL: 1 hour)
    V-->>APP: username: v-app-readonly-abc123<br/>password: random<br/>lease_id: db/creds/readonly/abc123<br/>lease_duration: 3600s

    Note over APP: Use credentials for 1 hour

    V->>V: Lease expires
    V->>DB: DROP ROLE "v-app-readonly-abc123"

Auth Methods

graph LR
    subgraph "Human Auth"
        LDAP[LDAP/AD]
        OIDC_M[OIDC/OAuth2]
        GH[GitHub]
        USERPASS[Userpass]
    end

    subgraph "Machine Auth"
        APPROLE[AppRole]
        K8S_AUTH[Kubernetes]
        AWS_AUTH[AWS IAM]
        GCP_AUTH[GCP IAM]
        JWT_AUTH[JWT/OIDC]
    end

    LDAP --> VAULT[Vault]
    OIDC_M --> VAULT
    GH --> VAULT
    USERPASS --> VAULT
    APPROLE --> VAULT
    K8S_AUTH --> VAULT
    AWS_AUTH --> VAULT
    GCP_AUTH --> VAULT
    JWT_AUTH --> VAULT

    VAULT --> TOKEN[Vault Token]
    TOKEN --> SECRETS[Access Secrets<br/>per policy]

    style VAULT fill:#e3f2fd
    style TOKEN fill:#e8f5e9

Implementation

Vault Setup and Configuration

#!/bin/bash
# Initialize Vault with Raft storage and auto-unseal

# vault server configuration
cat > /etc/vault.d/vault.hcl << 'EOF'
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3:8200"
  }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

api_addr     = "https://vault-1:8200"
cluster_addr = "https://vault-1:8201"

ui = true

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}
EOF

# Initialize Vault
vault operator init -recovery-shares=5 -recovery-threshold=3

# Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log

Dynamic Database Secrets (TypeScript Client)

import Vault from 'node-vault';

interface DatabaseCredentials {
  username: string;
  password: string;
  leaseId: string;
  leaseDuration: number;
}

class VaultDatabaseService {
  private vault: any;
  private activeLeases: Map<string, NodeJS.Timeout> = new Map();

  constructor(vaultAddr: string, token: string) {
    this.vault = Vault({
      apiVersion: 'v1',
      endpoint: vaultAddr,
      token,
    });
  }

  /**
   * Configure the database secrets engine.
   */
  async configureDatabaseEngine(
    connectionName: string,
    connectionUrl: string,
    allowedRoles: string[]
  ): Promise<void> {
    await this.vault.write(`database/config/${connectionName}`, {
      plugin_name: 'postgresql-database-plugin',
      connection_url: connectionUrl,
      allowed_roles: allowedRoles.join(','),
      username: 'vault_admin',
      password: 'initial_password',
    });

    // Rotate the root password so Vault controls it exclusively
    await this.vault.write(`database/rotate-root/${connectionName}`, {});
  }

  /**
   * Create a role that defines what credentials look like.
   */
  async createRole(
    roleName: string,
    dbName: string,
    creationStatements: string[],
    revocationStatements: string[],
    defaultTTL: string = '1h',
    maxTTL: string = '24h'
  ): Promise<void> {
    await this.vault.write(`database/roles/${roleName}`, {
      db_name: dbName,
      creation_statements: creationStatements,
      revocation_statements: revocationStatements,
      default_ttl: defaultTTL,
      max_ttl: maxTTL,
    });
  }

  /**
   * Get dynamic database credentials.
   */
  async getCredentials(roleName: string): Promise<DatabaseCredentials> {
    const result = await this.vault.read(`database/creds/${roleName}`);

    const creds: DatabaseCredentials = {
      username: result.data.username,
      password: result.data.password,
      leaseId: result.lease_id,
      leaseDuration: result.lease_duration,
    };

    // Schedule lease renewal before expiry
    this.scheduleRenewal(creds.leaseId, creds.leaseDuration);

    return creds;
  }

  /**
   * Renew a lease to extend credential lifetime.
   */
  async renewLease(leaseId: string, increment?: number): Promise<number> {
    const result = await this.vault.write('sys/leases/renew', {
      lease_id: leaseId,
      increment: increment ?? 3600,
    });
    return result.lease_duration;
  }

  /**
   * Revoke credentials immediately.
   */
  async revokeLease(leaseId: string): Promise<void> {
    await this.vault.write('sys/leases/revoke', {
      lease_id: leaseId,
    });

    const timer = this.activeLeases.get(leaseId);
    if (timer) {
      clearTimeout(timer);
      this.activeLeases.delete(leaseId);
    }
  }

  private scheduleRenewal(leaseId: string, ttl: number): void {
    // Renew at 2/3 of the lease duration
    const renewAt = Math.floor(ttl * 0.66) * 1000;

    const timer = setTimeout(async () => {
      try {
        const newTTL = await this.renewLease(leaseId);
        this.scheduleRenewal(leaseId, newTTL);
      } catch (error) {
        console.error(`Failed to renew lease ${leaseId}:`, error);
        // Application should handle credential refresh
      }
    }, renewAt);

    this.activeLeases.set(leaseId, timer);
  }
}

Vault Policies (HCL)

# policy: app-production
# Allows reading production secrets and generating database credentials

# KV secrets - read only
path "secret/data/production/*" {
  capabilities = ["read", "list"]
}

# Database dynamic credentials
path "database/creds/app-readonly" {
  capabilities = ["read"]
}

path "database/creds/app-readwrite" {
  capabilities = ["read"]
}

# Transit encryption
path "transit/encrypt/app-key" {
  capabilities = ["update"]
}

path "transit/decrypt/app-key" {
  capabilities = ["update"]
}

# Lease management (own leases only)
path "sys/leases/renew" {
  capabilities = ["update"]
}

# Deny access to other environments
path "secret/data/staging/*" {
  capabilities = ["deny"]
}

path "secret/data/development/*" {
  capabilities = ["deny"]
}

Kubernetes Auth Integration

import Vault from 'node-vault';
import fs from 'node:fs';

class VaultKubernetesAuth {
  private vaultAddr: string;
  private role: string;

  constructor(vaultAddr: string, role: string) {
    this.vaultAddr = vaultAddr;
    this.role = role;
  }

  /**
   * Authenticate to Vault using Kubernetes service account token.
   */
  async authenticate(): Promise<string> {
    // Kubernetes injects the service account token at this path
    const jwt = fs.readFileSync(
      '/var/run/secrets/kubernetes.io/serviceaccount/token',
      'utf8'
    );

    const vault = Vault({
      apiVersion: 'v1',
      endpoint: this.vaultAddr,
    });

    const result = await vault.write('auth/kubernetes/login', {
      role: this.role,
      jwt,
    });

    return result.auth.client_token;
  }

  /**
   * Create a Vault client with Kubernetes auth.
   */
  async createClient(): Promise<any> {
    const token = await this.authenticate();

    return Vault({
      apiVersion: 'v1',
      endpoint: this.vaultAddr,
      token,
    });
  }
}

Transit Secrets Engine (Encryption as a Service)

class VaultTransitService {
  private vault: any;

  constructor(vault: any) {
    this.vault = vault;
  }

  /**
   * Encrypt data using Vault's Transit engine.
   * The encryption key never leaves Vault.
   */
  async encrypt(keyName: string, plaintext: string): Promise<string> {
    const result = await this.vault.write(`transit/encrypt/${keyName}`, {
      plaintext: Buffer.from(plaintext).toString('base64'),
    });

    return result.data.ciphertext; // vault:v1:base64...
  }

  /**
   * Decrypt data.
   */
  async decrypt(keyName: string, ciphertext: string): Promise<string> {
    const result = await this.vault.write(`transit/decrypt/${keyName}`, {
      ciphertext,
    });

    return Buffer.from(result.data.plaintext, 'base64').toString('utf8');
  }

  /**
   * Rotate the encryption key.
   * Old ciphertext is still decryptable (key versioning).
   */
  async rotateKey(keyName: string): Promise<void> {
    await this.vault.write(`transit/keys/${keyName}/rotate`, {});
  }

  /**
   * Re-wrap ciphertext with the latest key version.
   * This does NOT decrypt — the plaintext never leaves Vault.
   */
  async rewrap(keyName: string, ciphertext: string): Promise<string> {
    const result = await this.vault.write(`transit/rewrap/${keyName}`, {
      ciphertext,
    });
    return result.data.ciphertext;
  }
}

Edge Cases & Failure Modes

Vault Unavailability

If Vault becomes unavailable, applications cannot fetch new secrets:

ScenarioImpactMitigation
Vault leader electionBrief write unavailabilityRaft consensus handles this (seconds)
Vault sealedAll operations blockedAuto-unseal with cloud KMS
Network partitionClients lose Vault accessLocal secret caching
Vault cluster downNo new secretsPre-fetched secrets, retry logic

WARNING

Never cache dynamic database credentials indefinitely. If Vault revokes a lease but your application uses a cached credential, the database may reject connections. Implement lease renewal and graceful credential refresh.

Seal/Unseal Failures

FailureCauseResolution
Cannot unsealLost key sharesUse recovery keys or rebuild
Auto-unseal KMS unavailableCloud KMS outageCross-region KMS, manual unseal backup
Seal migration failureSwitching seal typesFollow Vault migration guide carefully

Performance Characteristics

Vault API Latency

Operationp50p99Throughput
KV read1ms5ms20K req/s
KV write2ms10ms10K req/s
Transit encrypt3ms15ms15K req/s
Transit decrypt3ms15ms15K req/s
Dynamic DB creds50ms200ms500 req/s
PKI certificate20ms100ms2K req/s
Auth (AppRole)5ms20ms10K req/s
Auth (Kubernetes)10ms50ms5K req/s

Resource Requirements

Cluster SizeCPUMemoryStorageNodes
Development2 cores2 GB10 GB1
Small production4 cores8 GB50 GB3
Medium production8 cores16 GB100 GB5
Large production16 cores32 GB500 GB5

Mathematical Foundations

Shamir's Secret Sharing in Vault

Vault uses Shamir's Secret Sharing to split the master key:

where is the master key, are random coefficients, and shares are .

Reconstruction uses Lagrange interpolation:

With 5 shares and threshold 3, any 3 shares reconstruct the master key, but 2 shares reveal nothing.

Lease Duration and Security

The probability that a leaked dynamic credential is still valid when discovered:

For a 1-hour lease with 4-hour mean time to discovery:

The credential has already expired. This is the security power of dynamic secrets with short TTLs.

Real-World War Stories

War Story

Vault Auto-Unseal with AWS KMS — The Cross-Region Lesson

A company used AWS KMS in us-east-1 for Vault auto-unseal. When us-east-1 had a partial KMS outage in 2020, their Vault cluster could not unseal after a routine restart. All 200+ microservices lost access to their secrets for 4 hours.

Resolution: They implemented cross-region auto-unseal with a KMS key in both us-east-1 and us-west-2, with automatic failover. They also ensured all services cached their most recent secrets locally with a 1-hour TTL.

War Story

Dynamic Secrets Credential Storm

A company configured Vault's database engine to generate dynamic PostgreSQL credentials with a 1-hour TTL. During a deployment that restarted 500 pod replicas simultaneously, each pod requested new credentials, creating 500 PostgreSQL roles in under 30 seconds. PostgreSQL's pg_authid catalog table locked up, causing a cascading failure.

Resolution: They switched to a credential pool pattern — generating a smaller number of credentials with longer TTLs and sharing them across pods using a sidecar. They also staggered pod restarts to spread the credential generation load.

Decision Framework

When to Use Vault

Use Vault when:

  • You need dynamic secrets (short-lived database credentials)
  • Multi-cloud environment requiring unified secrets management
  • Encryption as a service (Transit engine)
  • Complex access policies with identity-based auth
  • PKI certificate issuance

Consider alternatives when:

  • Single-cloud deployment (use cloud-native secrets manager)
  • Small team without dedicated infrastructure operators
  • Simple secret storage needs (fewer than 50 secrets)
  • Cost sensitivity (Vault requires dedicated infrastructure)

Advanced Topics

Vault Agent Sidecar Pattern (Kubernetes)

# Vault Agent injector annotations for Kubernetes pods
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-db: "database/creds/my-app-readonly"
        vault.hashicorp.com/agent-inject-template-db: |
          {{ "{{" }}- with secret "database/creds/my-app-readonly" -{{ "}}" }}
          export DB_USER="{{ "{{" }} .Data.username {{ "}}" }}"
          export DB_PASS="{{ "{{" }} .Data.password {{ "}}" }}"
          {{ "{{" }}- end {{ "}}" }}
    spec:
      serviceAccountName: my-app
      containers:
        - name: my-app
          image: my-app:latest
          command: ["/bin/sh", "-c", "source /vault/secrets/db && ./start-app"]

Vault Performance Replication

For global deployments, Vault supports performance replication:

graph TB
    PRIMARY[Primary Cluster<br/>us-east-1<br/>Read/Write] --> SEC1[Performance Replica<br/>eu-west-1<br/>Read-only]
    PRIMARY --> SEC2[Performance Replica<br/>ap-southeast-1<br/>Read-only]

    SEC1 -.->|Write forwarded| PRIMARY
    SEC2 -.->|Write forwarded| PRIMARY

    style PRIMARY fill:#e8f5e9
    style SEC1 fill:#e3f2fd
    style SEC2 fill:#e3f2fd

Cross-References