Ever heard that you shouldn’t store secrets in your code, version control system, or even package them up in a configuration file?
The most common reasons for not distributing secrets via application artifacts include difficulty:
- maintaining proper access controls on artifacts, especially once retrieved from the artifact repository and copied onto innumerable filesystems
- rotating and expiring secrets
If you can’t publish application secrets inside application artifacts, how do you get secrets like passwords, api keys and certificates into your application at runtime?
The application (or a startup script) can query the application’s secrets from a secret store when it starts up. That secret store could be a file object, hardware security module, or other service.
But taking this one step further, how can your application prove to the system that stores these secrets that your application is authorized to read them so that it can continue starting up?
The most common way to implement authentication and authorization of process is via a shared secret, but we can’t do that here because there’s no way to share a secret to bootstrap this whole process. (Note: Systems Configuration Management systems solve this for low-entropy compute environments, but it’s messy once instances are disposable and cycling rapidly).
There seems to be a chicken and egg sort of problem where an application needs a secret in order to retrieve its secrets.
I call this The First Secret problem.
Trusted identity
Your compute platform must help you solve this problem by providing the application process a strong identity that the rest of the system can trust. The application can use the platform-provided identity to bootstrap itself with secrets and anything else that’s not appropriate to package with the application.
Remember, the reason we had a First Secret Problem is that we were trying to distribute a shared secret that could be used to authenticate the application process to retrieve the rest of the secrets. If the compute platform provides the application with a trusted identity, the initial shared secret problem disappears.
In AWS, your application compute process should launch with an execution role attached. An EC2 instance, container task, or Lambda function should launch with an EC2 instance role (profile), task execution role, or function execution role, respectively.
AWS’ deep integration between the compute platform and IAM provides your application with an identity. Now the application’s AWS SDK (or aws cli in a startup script) can use this role to generate short-term STS credentials by querying its metadata endpoint, and sign its requests to access other services with those credentials. Other Cloud providers have similar integration between compute and identity systems.
The short-term credentials acquired from the application instance’s identity can be used to retrieve secrets from SSM Parameter Store, Password Manager, or files that have been encrypted and stored in S3 with an appropriate access control policy.
When your application uses its execution role, it does not need (and shouldn’t have) an API Access Key for the compute provider’s platform. These same credentials can and should be used to access other services on the platform such as managed data stores and messaging services.
Stephen
#NoDrama
p.s. I’d love to hear if you’re challenged by The First Secret Problem or in manage secrets in the Cloud using a service like AWS Parameter Store.
p.p.s. If you have a configuration management system like Chef or Puppet, you could distribute secrets via a secure data bag that is encrypted with the shared secret established by the central server and the node on initialization. Of course, this doesn’t work great
