Secrets Management: HashiCorp Vault, AWS Secrets Manager, and Environment Variable Patterns

Editorial Team · · 4 min read

Stop Putting Secrets in Environment Variables

Well — stop putting them in .env files committed to Git, at least. Environment variables themselves are fine as a delivery mechanism. The problem is how teams manage the lifecycle: creation, storage, rotation, and access control.

A secret hardcoded in source code is obviously bad. A secret in an .env file tracked by Git is almost as bad — it's in every developer's clone, every CI cache, and every Git history forever. But even secrets injected as environment variables at runtime have issues: they're visible in process listings, crash dumps, and debug logs.

Let's look at proper secrets management from simplest to most sophisticated.

The Basics: What Counts as a Secret

Seems obvious, but teams regularly miss some of these:

  • Database credentials and connection strings
  • API keys (both yours and third-party)
  • OAuth client secrets and JWT signing keys
  • TLS private keys and certificates
  • Encryption keys and HMAC secrets
  • Webhook signing secrets
  • SSH private keys
  • Cloud provider credentials (AWS access keys, GCP service account keys)

If the value would be useful to an attacker, it's a secret. That includes internal service tokens that "only work on our network" — network boundaries get breached.

Cloud-Native Secrets Managers

AWS Secrets Manager

Stores secrets encrypted with KMS, provides automatic rotation for RDS and Redshift credentials, and integrates natively with Lambda, ECS, and EKS.

# Store a secret
aws secretsmanager create-secret   --name "prod/api/database"   --secret-string '{"username":"app","password":"xK9#mP2vL8","host":"db.internal"}'

# Retrieve in application code (Python)
import boto3
import json

client = boto3.client('secretsmanager')
response = client.get_secret_value(SecretId='prod/api/database')
creds = json.loads(response['SecretString'])

Cost: $0.40 per secret per month + $0.05 per 10,000 API calls. For most applications, this is negligible — a few dollars per month.

The automatic rotation feature creates a Lambda function that rotates credentials on a schedule. For RDS, it generates a new password, updates the database user, and stores the new secret — all without application downtime if you're using the staging label pattern.

GCP Secret Manager

Similar to AWS but with IAM-based access control at the secret version level. Pricing is slightly cheaper at $0.06 per 10,000 access operations and $0.06 per active secret version per month.

Azure Key Vault

Goes beyond secrets to manage certificates and cryptographic keys. The HSM-backed tier is required for certain compliance standards (FIPS 140-2 Level 2).

HashiCorp Vault: The Self-Hosted Option

When you need more control than cloud-native solutions provide — multi-cloud environments, complex access policies, or dynamic secrets — Vault is the standard choice.

Vault's killer feature is dynamic secrets. Instead of storing a static database password, Vault generates a unique, short-lived credential for each application instance:

# Configure the database secrets engine
vault write database/config/mydb   plugin_name=postgresql-database-plugin   allowed_roles="app-role"   connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/myapp"   username="vault_admin"   password="admin_password"

# Define a role with a 1-hour TTL
vault write database/roles/app-role   db_name=mydb   creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO "{{name}}";"   default_ttl="1h"   max_ttl="24h"

# Application requests credentials (gets unique short-lived creds)
vault read database/creds/app-role
# username: v-app-role-xK9mP2vL8
# password: A1b2C3-randomized
# ttl: 1h

Each application pod gets its own database user that expires after an hour. If credentials leak, the blast radius is one pod for one hour, not your entire database forever.

Vault Deployment Considerations

Running Vault in production is not trivial. It needs:

  • HA backend storage (Consul, PostgreSQL, or integrated Raft — I'd recommend Raft for simplicity)
  • Unsealing after restarts (auto-unseal with a cloud KMS key makes this operational)
  • Audit logging enabled to track who accessed what
  • Regular backup of the storage backend

If you don't have the operational capacity to run Vault, HCP Vault (HashiCorp's managed offering) or sticking with your cloud provider's secrets manager is the pragmatic choice. Self-hosted Vault that's poorly maintained is worse than AWS Secrets Manager that's well-configured.

Secrets in Kubernetes

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with kubectl access to the namespace can decode them. That's... not great.

Better options:

  • External Secrets Operator (ESO): Syncs secrets from AWS Secrets Manager, Vault, GCP Secret Manager, etc. into Kubernetes Secrets. The source of truth stays in the external secrets manager.
  • Sealed Secrets: Encrypts secrets so they can be safely committed to Git. Only the controller running in the cluster can decrypt them. Good for GitOps workflows.
  • CSI Secrets Store Driver: Mounts secrets as volumes instead of environment variables. Works with Vault, AWS, GCP, and Azure.
# ExternalSecret that syncs from AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-database-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-creds
  data:
    - secretKey: username
      remoteRef:
        key: prod/api/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/api/database
        property: password

Secrets in CI/CD Pipelines

CI pipelines need secrets to deploy, run integration tests, and push artifacts. Every major CI platform has encrypted secrets storage, but there are footguns:

  • Don't echo secrets in build logs. It sounds obvious, but set -x in a shell script will print every command including those with secret values.
  • Mask secrets in logs. GitHub Actions does this automatically for repository secrets. Jenkins requires the Mask Passwords plugin.
  • Limit secret access to specific branches. A PR from a fork shouldn't have access to production deployment credentials.
  • Use OIDC federation instead of long-lived credentials. GitHub Actions, GitLab CI, and CircleCI all support assuming AWS/GCP/Azure roles via OIDC tokens — no static keys to manage.
# GitHub Actions: OIDC federation with AWS (no access keys needed)
permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
      aws-region: us-east-1

OIDC federation is genuinely one of the best security improvements you can make to a CI/CD pipeline. No more rotating access keys, no more shared credentials across workflows. Each workflow run gets a short-lived token scoped to exactly what it needs.

Rotation Strategy

The hardest part of secrets management isn't storing secrets — it's rotating them without downtime.

The dual-secret pattern works for most scenarios: store both the current and previous version of a secret. Deploy the new version to your secrets manager, update application configurations to read the new version, then invalidate the old version only after all instances have picked up the change.

For database credentials, this means having two valid passwords simultaneously during rotation. For API keys, most providers support multiple active keys per account for exactly this reason.

Automate rotation on a schedule. 90 days is a common standard. For high-security environments, 30 days. For dynamic secrets via Vault, hours or even minutes.