Reading Time: 2.5 minutes
I’m getting internal reviews and putting the finishing touches on the Secret Delivery Research report, which I plan to publish this week.
Today I’ll try to help with one of the biggest challenge DevOps practitioners face when trying to deliver secrets to applications securely:
Engineering staff lack understanding of how to solve The First Secret Problem, especially using the respondent’s particular secret vaults and deployment platforms
Roughly 40% of respondents feel this way.
Since many people are deploying applications on VMWare with the aid of System Configuration management tools like Chef, let’s explore how to bootstrap trust with these tools.
Bootstrapping Trust with VMWare and CM Tools
The First Secret Problem describes how your compute platform or provisioning process must help you bootstrap the chain of trust for your application to retrieve its application secrets. Bootstrapping trust happens in two main ways:
- The compute platform has an identity mechanism built into it through deep integration with the virtual machine hypervisor and networking layer and makes platform managed identity available via, e.g. the AWS EC2 metadata endpoint
- Your system provisioning process uniquely identifies each machine when it is first provisioned and securely integrates that machine into your configuration management process
The second approach is usually easier said than done, but is important for those maximizing the security and value of existing virtual machine cluster investments.
Here’s a quick sketch of a process to bootstrap trusted identiy using Terraform, VMWare and Chef:
- Start a trusted, privileged execution environment with authority to provision hosts; ideally an automated deployment server, but often an operator’s local machine
- Use Terraform VMWare vSphere provider to provision a new virtual machine
- Use a Terraform
local
provisioner to bootstrap the new virtual machine with the Chef agent and authorize its registration with the Chef Server usingknife bootstrap
- Generate and install the list of recipes to run on first boot on the new machine
- Force
chef-client
to run on the new machine to complete system configuration using, e.g.knife ssh chef-client $node_name -j /path/to/first-boot.json
Chef describes steps 3-5 well in their documentation for bootstrapping a new node, particularly the “Bash User Data” section. Similar approaches exist for Ansible, Puppet, and other CM systems.
There are all sorts of variations on this basic process that depend on an organization’s specific network, system authorization, and security policies. You will probably want to include certain elements that are foundational to trust in the virtual machine template. Examples of artifacts you may want to bake-in to the base template are the TLS certificates the system will trust and and the public key of the local user needed to bootstrap the machine in step #3.
Now:
- Chef can manage the installation and distribution of application secrets or material used to retrieve application secrets, e.g. credentials to access an Enterprise secret vault like Thycotic Secret Server or SafeNet Luna or an API Key for AWS/Azure/GCP
- The root of trust a has been pushed back to the deployment system and Chef server and the inputs to those systems
- Privileged access the new machine is
- defined by what is defined in the delivery pipeline and system configuration management code
- controlled by whomever has access to change delivery pipeline and configuration management code
Be sure to configure appropriate access controls for:
- source code repositories
- automated delivery jobs and secrets managed by CI/CD systems
- the configuration management server
I’d love to hear from you if you have questions about The First Secret Problem, bootstrapping machine identity, or the chain of trust. Hit reply!
#NoDrama