DevOps
Building a business with continuous delivery and Serverless
Hey – I’m back! I’ve been heads-down learning what Cloud teams need to deliver to AWS securely and make IAM usable, and building that into a business delivered via SaaS & infrastructure code libraries: k9 Security. I’d like to share my experience in how continuous delivery and serverless supported that endeavor. I hope this experience
Never felt more DevOps than when I was Serverless
Serverless helped me integrate DevOps principles into my application delivery workflow like never before.
DevOps interview questions: Organizational Maturity
What questions should engineers ask in a DevOps interview to understand the organization’s DevOps maturity? These questions should not be about specific tools and platforms. They should be about people, processes, how we get work done, and how we collaborate. This post is written primarily for candidates. Hiring teams can also use these questions to
Empathy in Action
What the world needs now is empathetic action by those with power.
Guide to Tagging Cloud Deployments
Resource tagging is the primary context communication method in the Cloud but many organizations struggle to define the terms and a model to describe and analyze their Cloud deployments. The k9 Security guide to tagging cloud deployments will help you define your tagging standard and explain why it’s important.
Converting energy into something that matters
DevOps practitioners can help deliver value to customers quickly and collaborate across teams, even or especially, in times of crisis
Help with big changes
I hope that you and your family are safe and healthy. COVID-19 is forcing many organizations to change long-standing policies and processes very quickly under great uncertainty. I’d like to help you navigate through those changes, especially if you support healthcare services. First, there are resources in the Knowledge Base that you may find helpful
RSA Conference 2020: A DevOps Perspective
I attended two and a half days of the RSA security conference last week and I’d like to share my first-timer perspective with you. Summary The opening keynote challenged Information Security to reorganize to collaborate better with users, business, risk, and IT teams. My post-conference perspective is that this challenge is largely ahead of us.
Computing a Risk Estimate using Netflix’s riskquant
This post computes a realistic annual loss estimate in dollars for an ecommerce application using the riskquant tool that models the distribution of possible impacts and probabilities appropriately.
Modeling Risk in Cloud Deployments
Information Security risks are those risks “that arise from the loss of confidentiality, integrity, or availability of information or information systems and reflect the potential adverse impacts to organizational operations, organizational assets, individuals, other organizations, and the Nation (NIST 800-30).” Risk assessments should help people understand a given Cloud deployment’s information security risks and help
Modeling Security in Cloud Deployments
Cloud deployments often use tagging to describe the context of a compute or resource such as a who owns or what application a virtual machine or object storage bucket belongs to. However, the common resource tagging models in use don’t describe the context required for people or tools to assess security or manage risk easily.
Research: Problems Engineers have Securing Cloud Deployments and ‘Shift Left’
I hit an obstacle on my way to extending the common resource tagging models with context to describe security and risk attributes. I couldn’t get past that while I feel this should be a settled engineering practice, it isn’t. Suspecting bad assumptions lurking around, I spoke to some engineers practicing DevOps with titles like DevOps
Research: Problems with top free security assessment tools (2020q1)
I’m researching how engineers assess the security posture of their Cloud deployments and evaluate risk to those deployments so they can improve it. Reading Time: 10 minutes The research starts with these questions: What’s the hardest part about assessing and improving the security of your Cloud deployments? When was the last time you tried or
Continuous Delivery with Serverless, CircleCI, and Lambda: The Good, Bad, and the YAML
Our Serverless continuous delivery pipeline lets us achieve Elite levels of Software Delivery Performance.
Minimize your delivery pipeline and platform dependencies
To escape The DevOps Organization Build Trap, I suggested that you minimize the distinct sets of components your organization uses to deliver and operate software. Those components represent a dependency tree and are integrated, enhanced, and worked-around by your team to meet your organization’s requirements on your dime. While all delivery pipelines and deployment platforms
Help for your Agile or DevOps Resolutions
Happy New Year! Many of you have freshly resolved to be more agile, adopt DevOps, or ramp-up your software delivery improvement efforts. The NoDrama Agile and DevOps theme page will help you accomplish these resolutions by: building a solid understanding of what DevOps is, why it works, and the value it provides (spoiler: DevOps is
Debugging AccessDenied in AWS IAM
Ugh… that looks like it could be the start of a two hour or two week long goose chase. Understanding why access was denied and implementing a secure solution can be complicated. Sometimes it’s not even clear where to start and what to do when you get stuck. Update: An improved version of this Debugging
Site Operations Blessing
Here’s a little blessing you can say for your teams and website responsible for operations this peak-filled holiday season: Thank you to all who have improved our system’s capabilities and robustness for our customers. May our dashboards be green and charts stable May fixes flow swiftly through the pipelines to production May instances start quickly
Review: The Unicorn Project
You can learn a lot from The Unicorn Project about the day-to-day leadership, management, and ‘manage-up’ skills to transform yourself, a team, or an organization using The Five Ideals.
Why is AWS IAM So Hard?
Why is AWS IAM so @!#^$!# hard? One of my favorite Directors of Cloud Platform AWS Identity and Access Management (IAM) is a security tool that controls access what AWS API actions that Principals (roles, users) are allowed to perform on which AWS resources: an S3 bucket, EC2 instance, DynamoDB table, etc. IAM is really
Changing organizations successfully
Organization and process changes are full of challenges. First, the existing structure and process is (probably) a ‘stable state’ that people are comfortable with and whose very position in ‘the machine’ reinforces. Even if no one likes the existing state. Second, there are forces within the organization that will actively oppose the change because people
The DevOps Organization Build Trap
I am learning a lot from Melissa Perri’s ‘Escaping the Build Trap’ in my studies of product management. Perri says in chapter 9 ‘Organizing Your Teams’: You can’t build an organizational structure without a product vision, because the value streams are not apparent. Perri, Melissa. Escaping the Build Trap: How Effective Product Management Creates Real
Improving software delivery processes at the organization level
Today I will share my high-level process and questions for reflecting and improving an organization’s delivery process and outcomes. First, review the Aspects of Software Delivery Performance (DORA) that generate improved business outcomes: Deployment frequency Lead time for changes Time to restore service Change failure rate Second, imagine what the ideal delivery process looks like
Minimum Viable Pumpkin
I’m sharing this story to illustrate how you can frame, approach, and explain all kinds of activities as Agile delivery processes. We decompose problems and their solutions all the time and Agile/Lean is applicable for more than just software — in this case a pumpkin.
Read raw, unedited customer feedback
Here’s an idea to help your team build empathy for what customers are feeling when they use your service. Read raw, unedited customer feedback about your product or service as a team. There’s probably a ton of this data kicking around your organization’s customer service, chat, CRM tools, and social media. Often this data is
Uninterrupted Time
When was the last time you worked uninterrupted for one hour? An entire hour. Much of the work technologists and other knowledge workers perform requires significant time to load context, think, and work out the solution. But frequently, we do not give people that time. One of the most important transformations of my career happened
Docker cpu resource limits
Docker allows you to constrain the CPU resources available to a container on either a relative or absolute basis — this post shows you how and why you might do that.
Docker memory resource limits and a heap of Java
Ok, so maybe you’re convinced why you need to set limits for container resource usage, especially when running on shared container application platforms like ECS, Kubernetes, and Swarm. In this post, I’ll show you how to set memory limits for a basic Java web app built with Spring Boot. Most container clusters are memory bound,
The importance of Docker container resource limits
This post explains why enforcing resource limits is a critical best practice for running workloads on shared compute platforms such as a container cluster orchestrated by ECS, Kubernetes, or Swarm.
The design process
This post describes QualiMente’s design process. This design process incorporates feedback from authors, a design review, and (optionally) prototypes. Once design feedback is collected, the delivery team and product manager have much more knowledge about whether to continue, how to approach the solution, and how to break up and sequence the work. Additionally, you can communicate to collaborating teams with the confidence that you’e probably identified half or more of the design’s defects.
Zzzt! Closing the loop
Reading time: 90 seconds In QualiMente’s Secure Cloud team retrospective on Friday, we identified a couple of improvement actions: For context, we are several weeks into building a new service called k9 that helps people and access analysis tools understand “who has access to what” in AWS. As product owner and team lead, I prioritized
Feedback loops are a beautiful thing
Delivery process improvement ideas rise from the frustration felt by teams executing the process. If you want to improve a team’s delivery processes, I recommend putting your team in control of improving itself and support the team with practices and resources to help them do that.
State of Application Secret Delivery and Audit Practices – 2019
The State of Application Secret Delivery and Audit Practices should help you explain to others why delivering application secrets is challenging, the risks of unsafe practices, and a general approach for improving your application secret delivery process.
Concept: Configuration
Configuration is data interpreted by a program to change its behavior so that it supports a particular use case. This post describes the challenges and techniques for creating a configuration interface that is safe and usable.
Can you handle 10x or 100x more application components?
Reading Time: 2 minutes Many organizations are making fundamental changes to the way their applications are structured and operated. These changes are intended to address real problems. Structurally, monolithic applications can severely hamper change delivery rate as the regression test process comes to dominate change delivery time or changes from many teams being held up
Bootstrapping Trust – Terraform+VMWare+Chef
Lack of understanding how to deliver secrets to applications securely is the biggest challenge identified by DevOps practitioners in the Secret Delivery and Audit Practice study. This post describes how you can deliver secrets securely when using VMWare, Terraform, and Chef.
Learning to Debug
Finding the needles in a large haystack is sometimes the hardest part of solving a problem. Debugging is also a skill you can learn and develop.
What are application containers and what problems do they solve?
This post introduces application containers and explains how Docker helps you package and run (crazy) applications inside neat little boxes, keeping the hosting computer nice and tidy.
Building Security Skills for an AWS Cloud Migration
The Security skills needed for a Cloud migration may be in shortest supply and the most difficult to acquire. This post describes resources you can use to build Security skills throughout your engineering team and to integrate your Security Engineers with greater leverage.
The Prime Directive of Retrospectives: Safety
(Reading Time: 1min) If you feel like you’re not getting good participation and feedback in your sprint or project retrospectives, you’ve got a problem. That problem might be that people don’t feel safe in sharing what they really think and feel. Tactically, if people aren’t sharing the information you need to improve continuously, it’ll take
Why I Design
(Reading Time: 2.5 minutes) I design to avoid waste. Waste in building something: nobody wants can use effectively can afford That is, design is a process that helps maximize the value of systems we create. The design process is also critical to understanding whether we should build them at all. Some systems will have low
The Future of Information Security is Simpler (I Hope)
We can improve the world’s software by transforming ‘complicated’ and ‘complex’ into something simper, even boring. This transformation isn’t easy but there is a way to commoditize security and other specialty practices that make your system safer and more reliable.
Converging Infrastructure to a Desired state
This post describes the process infrastructure as code tools like Terraform use to ‘converge’ the real world to the state you declared in your code.
Concept: Infrastructure as Code
When you adopt “Infrastructure as Code” (IaC), you manage infrastructure by defining the desired state of your infrastructure in source files and use a tool to realize those infrastructure resources.
Like nails on a chalkboard: ‘We haven’t shipped’
Recently, the Hertz car rental company sued Accenture for $32M in damages. The suit (pdf) alleges breach of contract in Accenture’s agreement to build and deliver a new website. This new site was supposed to: support the Hertz, Thrifty, and Dollar brands, starting with Hertz, North America be a ‘responsive’ site that works across phone,
Controlling Access to Secrets within CI/CD Systems
Controlling access to secrets within Continuous Integration and Delivery (CI/CD) systems is very important. CI/CD those systems often use credentials with highly-privileged access to create and publish business critical artifacts and application deployments. This post shares pointers for how to protect access to secrets in Jenkins, Gitlab Enterprise, CircleCI, and CodeFresh.
Origin Story: The Blue-Green Deployment Method
Ever wonder what the heck the ‘Blue Green’ deployment method is or how it got its name?
Heartbleed’s lesson about delivering change
Investing in robust delivery processes to lower the minimum time for an idea to go to concept to customer is not at odds with security, ‘stability’, and customer happiness. In fact, a robust delivery pipeline is a mechanism by which you can achieve all of these.
The First Secret Problem
Your application’s compute execution role is its identity in the Cloud. Use it to bootstrap access to runtime secrets in addition to authenticating to other provider-managed services.
APIs are the ante for the Cloud service game
If a service can’t be provisioned and managed via an API, it’s not a Cloud service.
What is a Cloud?
The Cloud changes the economics of application deployments by shifting to pay-for-what-you-use consumption based model. You can and should use the Cloud’s resource provisioning APIs to allocate resources when you need them and release them when you’re finished to create an economical deployemnt.
Upgrade your testing process for Continuous Delivery
If you are thinking of adopting continuous delivery, now is the time to evaluate the effectiveness of your verification process and identify opportunities for improvement in safety, efficiency, and throughput.
Upgrade your deployment process to support Continuous Delivery
Once you’ve automated the core of your deployment process and decoupled ‘deploy’ from ‘release’, the majority of concern about adopting CD should melt away.
Foundations of Continuous Delivery: Deployment and Test Automation
Once Continuous Integration is in place, you can focus on improving your project’s delivery process. Today I want to highlight the need for two foundational of a continuous delivery process: deployment and test automation. Average Read Time: 2 minutes Recall that Continuous Delivery is a practice that: ensures our code is always in a deployable
Good release artifacts drop onto deployment platforms
This deep-dive on Continuous Integration started with saying that a project’s CI process should build the canonical release artifacts for a project or product. These release artifacts should archive the application that was built along along with application-specific tools such as scripts required to run the application. Avg Reading Time: 3 minutes These artifacts will
Continuous Integration of a Containerized Application
Learn how to create a continuous integration process that packages custom software into a Docker image, tests, and publishes that as a releasable artifact.
Continuous Integration for Infrastructure Code
Continuous Integration can and should be applied to infrastructure code written in tools like Terraform and CloudFormation. This post describes how.
What does Continuous Integration look like in 2019?
The primary goal of a continuous integration process is to provide rapid feedback on changes to the development team. Teams use CI to determine whether a change is reasonable to deploy for further testing.
Improving Software Delivery Performance with Continuous Integration and Continuous Delivery
Much of the extra effort and delay in software delivery processes is hiding in long-lived feature branches and test environments built from long-lived release branches. Deliver more frequent releases with lower lead times using Continuous Integration and Delivery (CI/CD).
Adopting a Robust Incident Response Process
You don’t need to build an incident response process from scratch, adopt!
A Good Response for an Incident in a Complex System – GitHub
What does a good response look like for an incident in a Complex system? Start by establishing a system for classifying incidents in terms of customer and business impact. Then establish a process to manage the restoration of service that prioritizes the following external goals: safety for the service’s customers and their data restoration of
Are Target Resolution Times for Incidents Harmful?
Incident resolution times are a lagging indicator of team and organizational performance, not constructive guidance during incident response.
Cooking Something Good with DevOps
This is the final piece of a three-part series introducing the Principles and Value of DevOps. I am excited and relieved that after 15 years of trying to discover, learn, apply, and teach these practices, that we have matured so that it’s clear: what principles we should apply how to guide the application of those
The Value of DevOps to Your Organization
DevOps provides value to organizations by enabling them to change quickly and safely.
Principles of DevOps
DevOps will help you optimize the delivery of services that are built on technology _and improve the services_ themselves.
Ready, Set, Eliminate Drama!
Life moves pretty fast. If you don’t stop and look around once in a while, you could miss it. — Ferris Bueller Welcome to the No Drama DevOps Daily blog! We’re going to learn how to design, build, and operate modern technology systems in the Cloud safely and efficiently, one day and concept at a time. Each day,