What questions should engineers ask in a DevOps interview to understand the organization’s DevOps maturity? These questions should not be about specific tools and platforms. They should be about people, processes, how we get work done, and how we collaborate.
This post is written primarily for candidates. Hiring teams can also use these questions to communicate the good stuff they’re doing. Questions posed by experienced consultants and engineers in my network inspired this train of thought.
Bret Fisher got me thinking by asking ‘What questions do you ask in an interview to understand the orgs DevOps maturity?’:
Some people may view organizational maturity as an advanced topic, but I think it’s the place candidates should start.
People suffer more from bad culture, management, and processes than bad tooling. Good cultures will address bad tooling.
Many interview processes are built around technical topics that reveal little about what it’s like to work there. They’re trying to determine if and how you fit into their system as a technical contributor. Those organizations may not be bad.
But a candidate needs to understand the inverse of the relationship.
How will I be managed and what is that going to feel like?
Ask these questions and find out:
- How do your delivery and operational processes contribute to the success of the business? How do you measure those processes?
- Can you tell me about the last time your team improved its daily work based on issues identified in your retrospective process?
- What incentivizes product managers to build sustainable, reliable systems?
These questions and discussion should:
- reveal a lot about the organization’s maturity
- be answerable in 3-5 minutes each by anyone with Senior or above in their title
Let’s discuss why each of these questions are important and learn what you need to make good choices about your next team.
Objectives
Q1: How do your delivery and operational processes contribute to the success of the business? How do you measure those processes?
Asking how delivery and operational processes contribute to the organization will help you understand how critical technical functions are measured, perceived, valued, and managed across the organization. This reveals how your prospective department interacts with other departments.
Good answers should reference measures similar to the Aspects of Software Delivery Performance (Forsgren): change delivery throughput, minimum latency of change, change failure rate, time to recovery.
Maybe you’ll hear a well-grounded story of how planning and delivery has improved over the past year, perhaps triggered by large program slip or operational incident.
This is a good answer in narrative form:
“A year ago, we were lucky to deploy monthly. Since we did
TECH_THING_X, we’re able to deploy daily on-demand if needed. However, most teams deploy weekly. That delivery capability helped the business deploy our app’s new remote work features that is helping grow our business during COVID-19. In fact, that’s why we’re hiring 🙂
Or you may hear a direct description of the KPIs or OKRs used by the team and how those are fed to other functions like product management and influence how work is done.
“We use Agile/Scrum/Kanban,” is not enough. If your prospective leadership answers this way, you can prompt a clarification with something like, “How do you know the work you’re doing improves capabilities that matter to the business? How do you prioritize work?”
Anyone in your interview loop in a management role or with ‘senior’ in their title should be able to explain how their delivery and operational processes enable the business to execute and scale. That’s a responsibility of their role. They should be forthright with the problems and how they are being improved. We all have work to do.
Bad answers will sound like:
- Uncertainty: Inability to explain the delivery and operations capabilities in terms everyone can understand. This should be second nature for tech leaders and permeate their daily work and conversations: process development, architecture, backlog grooming and prioritization, etc.
- Myopia: An avalanche of technical detail, tools, or procedural ceremony. I understand y’all are busy, but what are you trying to accomplish?
- Deflection: They may change the topic to something else, say “you don’t need to worry about that,” or claim they cannot talk about it. They may not even realize they’re doing this. This may mean it’s so bad they are uncomfortable talking about it. Or it could be a pathological organization wherein you’re interviewing to be an order-taker.
- Inconsistency: Inconsistent answers between leaders in your interview loop. People don’t need to use the exact same words. However, the same measurements should be referenced or they’re not really department-level measurements that align people to the mission.
This question is so important I recommend weaving it into at least two conversations with senior people in the interview loop.
Now let’s drill into the inner workings of your prospective team.
Continuous Improvement
Continuously improving the team’s daily work is critical to increasing the long term happiness and capacity of the team. Gain insight on whether you’ll be happy in that team with a question like:
Q2: Can you tell me about the last time your team improved its daily work based on issues identified in your retrospective process?
Everyone in the interview loop should be able to describe recent improvement activities.
The improvement activity doesn’t need to be a huge project and probably shouldn’t. Improvement activity needs to be frequent and meaningful instead of bearing the pain until people are about to quit.
The team member might recall fixing a bug in a tool so that deployments are safer. Or maybe creating a runbook to codify an operational activity. Then ask them to describe how the team identifies and schedules improvement work.
This borders on being a “loaded question” because it assumes the team uses some sort of retrospective process to gather feedback and improve its work. This practice shouldn’t be controversial, but it’s often neglected in practice.
I suggest making systematic continuous improvement a requirement in your next team. Having no official agency to fix problems you encounter is soul-crushing and unnecessary.
Next, drill into a common problem for DevOps practitioners: unsustainable systems.
Sustainability
My friend Paul Bauer builds and operates large scale systems. He (separately) asked an important question about system sustainability and incentives:
Q3: What incentivizes product managers to build sustainable, reliable systems?
Paul’s question is a great one and your interview loop should answer it.
Product managers must balance several competing priorities:
- Customer satisfaction
- Lifetime customer value
- Product profit & loss
- Product & Operations team satisfaction
A product manager must balance these over the medium and long term for the product to achieve and sustain success. This is a difficult job. Managing complex, dynamic systems requires frequent adjustment through a probe-sense-respond loop.
The first three priorities have well-defined, frequent methods of measurement and powerful advocates with separate management chains.
Product and Operations team satisfaction is also critical to the success of a growing product. Without those teams, the product cannot adapt to customers changing needs and won’t be there for customers to use when they need it. But team satisfaction or health is harder to measure.
Product Managers will overload Operations and Product teams without robust feedback. This does not mean the PM is a bad person. More probably, the organization lacks well-defined measurements, controls, and incentives that transmit feedback across functions, aka silos. And they are driven by the other well-defined measures.
So listen carefully for how feedback is transmitted across departments and management chains.
Here are mechanisms organizations can use to build sustainable systems. They are grouped by time and organizational scale:
Small:
- Retrospectives that uncover why work succeeded or failed
- ~20% of time dedicated to improving daily work
- PMs join Sev1 incidents & post-mortems and observe
- Align bug severity levels to the severity of incidents they’ve triggered; Sev1 incident should have Sev1/Critical bugs
- Sev1/Critical bugs are automatically
- Engineering team lead has authority to say ‘no more features until…’
- Compensate incident response time outside of normal working hours two for one
- Require people use time off
Guiding principle: We understand that unexpected bad stuff happens and we’ll deal with it in the short term, but we also expect to improve the system to eliminate those problems over time. Don’t wake me up to put a bandage on something we don’t plan to fix next week.
Medium:
- Operations Reviews should include incidents, heroic efforts, trends for open and resolved Sev1/Critical issues, recent improvements to daily work
- “360 degree feedback” and internal team satisfaction impacts compensation and promotion to next level for Product Management, Product Development, and Operations (yes this is hard, but Directors gotta direct)
Guiding principle: Management and leaders are regularly measured and held accountable for team satisfaction and health.
Long:
- Internal mobility for Product Management Product Development & Operations Engineers
- Exit Interviews and Personnel change post-mortem
Guiding principle: People need to be able to vote with their feet and managers should listen when they do.
I recommend weaving this question into conversations with at least two people in your interview loop, covering both individual contributors and managers.
Closing
This post provided three questions you can use in interviews to gain insight into how, why, and which work gets done. This information is likely critical to your future happiness.
These questions signal to interviewers that you expect work to be managed well and your commitment valued.
These questions also provide a way for teams to share how they embody DevOps principles. Sharing what you’ve done and how you’re improving against each of these questions is a great differentiator.
Stephen
#NoDrama
