Bootstrapping Trust – Terraform+VMWare+Chef

by | Jul 8, 2019 | DevOps, Infrastructure As Code, Secrets | 0 comments

Reading Time: 2.5 minutes

I’m getting internal reviews and putting the finishing touches on the Secret Delivery Research report, which I plan to publish this week.

Today I’ll try to help with one of the biggest challenge DevOps practitioners face when trying to deliver secrets to applications securely:

Engineering staff lack understanding of how to solve The First Secret Problem, especially using the respondent’s particular secret vaults and deployment platforms

Roughly 40% of respondents feel this way.

Since many people are deploying applications on VMWare with the aid of System Configuration management tools like Chef, let’s explore how to bootstrap trust with these tools.

Bootstrapping Trust with VMWare and CM Tools

The First Secret Problem describes how your compute platform or provisioning process must help you bootstrap the chain of trust for your application to retrieve its application secrets. Bootstrapping trust happens in two main ways:

  1. The compute platform has an identity mechanism built into it through deep integration with the virtual machine hypervisor and networking layer and makes platform managed identity available via, e.g. the AWS EC2 metadata endpoint
  2. Your system provisioning process uniquely identifies each machine when it is first provisioned and securely integrates that machine into your configuration management process

The second approach is usually easier said than done, but is important for those maximizing the security and value of existing virtual machine cluster investments.

Here’s a quick sketch of a process to bootstrap trusted identiy using Terraform, VMWare and Chef:

  1. Start a trusted, privileged execution environment with authority to provision hosts; ideally an automated deployment server, but often an operator’s local machine
  2. Use Terraform VMWare vSphere provider to provision a new virtual machine
  3. Use a Terraform local provisioner to bootstrap the new virtual machine with the Chef agent and authorize its registration with the Chef Server using knife bootstrap
  4. Generate and install the list of recipes to run on first boot on the new machine
  5. Force chef-client to run on the new machine to complete system configuration using, e.g. knife ssh chef-client $node_name -j /path/to/first-boot.json

Chef describes steps 3-5 well in their documentation for bootstrapping a new node, particularly the “Bash User Data” section. Similar approaches exist for Ansible, Puppet, and other CM systems.

There are all sorts of variations on this basic process that depend on an organization’s specific network, system authorization, and security policies. You will probably want to include certain elements that are foundational to trust in the virtual machine template. Examples of artifacts you may want to bake-in to the base template are the TLS certificates the system will trust and and the public key of the local user needed to bootstrap the machine in step #3.

Now:

  • Chef can manage the installation and distribution of application secrets or material used to retrieve application secrets, e.g. credentials to access an Enterprise secret vault like Thycotic Secret Server or SafeNet Luna or an API Key for AWS/Azure/GCP
  • The root of trust a has been pushed back to the deployment system and Chef server and the inputs to those systems
  • Privileged access the new machine is
    • defined by what is defined in the delivery pipeline and system configuration management code
    • controlled by whomever has access to change delivery pipeline and configuration management code

Be sure to configure appropriate access controls for:

I’d love to hear from you if you have questions about The First Secret Problem, bootstrapping machine identity, or the chain of trust. Hit reply!

#NoDrama