I synthesized and expanded what I’ve been sharing about delivering secrets to applications into a 1 hour talk and presented it at the Phoenix DevOps meetup on Wednesday.

This talk covers (Recording (YouTube), Slides (Google)):

  • the fundamental challenges of handling secrets, including The First Secret Problem
  • common ways you can lose control of them in delivery processes
  • how to deliver secrets to your applications in a secure and highly-available manner…from within your existing delivery pipeline
  • patterns and anti-patterns for organizing secrets
    • Anti-pattern: Global Secrets in Target Vault
    • Pattern: Version Secrets in Target Vault
  • ways to consume secrets inside applications: environment variables, files, and direct to the Vault API

We worked through this delivery process end to end:

Secret Delivery Integrated into Application Delivery

I’ll dig into the topics we haven’t covered on the list soon.

Here’s the summary…


  • Keep secrets out of source control unless they’re encrypted
  • Distribute secrets to a Target Vault in the application runtime environment
  • Leverage application’s platform provided identity to retrieve secrets
  • Prefer reading secrets from access-controlled files on in-memory filesystem

If you want to get started improving your own process

  • Start by looking at Mozilla’s SOPS tool and understand its capabilities for encrypting files with secrets so they can be stored inside source repositories safely
  • Several tools have emerged to support last-mile secret delivery on Kubernetes: Soluto’s Kamus, Bitnami’s Sealed Secrets, Helm Secrets

If you would like me to assess your existing secret delivery process and suggest improvements, reply to this email and we’ll get it done at your convenience. First three people to reply get their assessment for free, the rest for a low price that you can expense if you hit me up by Friday, May 10.