Perhaps even more so than science, engineering is akin to writing or painting in that it is a creative endeavor that begins in the mind’s eye and proceeds into new frontiers of thought and action, where it does not find so much as make new things.
Petroski, Henry. The Essential Engineer, Kindle Location 3008
Reading Time: 5 minutes
Designing a solution is one of engineering’s primary creative activities. We use our knowledge of principles, techniques, and primitives from science, engineering, and industry to create a solution to a problem.
If that problem is new or unique, we may need to invent a novel solution from whole cloth. If the problem is similar to something that has been done before, we may be able to configure or tailor existing technology to our needs. And of course there’s a whole lot of space in between.
First, we must define the problem that needs solving so that we attempt to solve the ‘right’ problem. I start by writing a few sentences describing the problem and how it impacts customers, the business, or organization. Constraining the problem definition to a one to three sentences helps focus on the most essential needs of the system. This may expand later, but resist that temptation at the start. You will refine the problem statement as you learn, anyway.
This short form of problem statement is most appropriate for a feature or sub-system of an existing product or service. You will need a more robust approach for defining ‘the problem’ if you are designing a new product or family of product features. In that case, I suggest trying a Lean ‘business model canvas’ approach.
Once you have a good initial understanding of the problem, you can start designing a solution to the problem. If you don’t have a good enough understanding of the problem to design a solution directly, your next step should be to design something which helps you move the problem and solution definition process forward.
The solution sketch
The solution starts taking shape from a sketch. First the solution exists in our mind, then usually a sketch on a piece of paper, whiteboard, or tablet. At this point, the solution and problem definitions are messy and changing rapidly. Let the ideas flow and the system evolve as you consider different alternatives for implementing the system and what the solution’s essential form might be.
My go-to sketch is a block-ish diagram depicting the system’s main actors and logical architecture. Here’s a sketch for k9, a system QualiMente is building to help people determine “who has access to what” in AWS:
The most important elements of this diagram are: the people, problems, flow of data, and information provided to users.
Here are some principles and questions I use to guide and inspire my own design creation process:
Solve the right problem
The biggest risk I’m trying to manage at this point is solving the wrong problem or a problem that no one cares about.
To help with this, I start by putting the people and their needs in the sketch. Often, I can pull this information from a Lean Canvas or Wardley Map built with customer input. This will be a constant reminder that the solution serves people. Second, those people prompt me to refine the problem statement as I go.
Key questions:
- Does persona ‘X’ really have this problem?
- Will this solution help them? How much?
I imagine the information these people need to make decisions or what they want to produce and make quick mockups of those artifacts. These represent key outputs of the system that users are meant to interact with. These mockups can expand into paper prototypes you can use to generate real feedback from potential users, whether in the form of a UI, report, or api usage guide.
For example, one of the first things I did for k9 was mockup the ‘access inventory’ report in json so that I could show it to a few people and gather feedback. Concurrently, we also started researching if and how we would get all the data in the report.
Learn from existing solutions
The world produces, tests, and validates solutions in a massively parallel fashion. We can jumpstart our own designs by learning from existing solutions and previous experience.
Key question: How do people address this problem now? What are the gaps?
We may be able to survey the technology literature and market to find components or services that solve huge portions of the problem statement. At a minimum, that survey can inform of important limitations and common design choices. In the case of k9 we were able to draw on much of the knowledge learned researching Secret Delivery and Audit practices.
Key question: How do you solve this problem without ‘modern’ technology?
For example, merchants, shippers, and bankers worked out how to buy, sell, transport, and insure goods across oceans long before relational databases, elastic cloud computing, and blockchain.
Many core practices of accounting, insurance, and (physical) security are hundreds of years old (or older) and offer great insights into handling data loss, communication problems, and other failures of distributed systems using pen and paper. You may choose to integrate concepts and processes such as bills of material, reconciliation, and audit logs into the system. These time tested concepts may be useful in either the main computation process or as control or repair mechanisms.
Simplicity
The design “should be as simple as possible, but no simpler. (1)”
Simplicity is a core principle of building reliable and secure systems, so this is the primary ‘engineering’ guidance I observe at this point. This helps drive out accidental complexity, leaving focus and error budget for the essential complexity that remains (c.f. No Silver Bullet).
Of the hopefully essential complexity that remains, I try to push as much complexity as practical from operational domains into upstream development, test, and deployment processes.
Key Questions:
- What’s the simplest thing that could possibly work?
- What could I remove or implement with a simpler technology and still satisfy the user’s needs?
I suggest you keep pruning until you can tell that customer-visible features cannot be provided or system expectations cannot be met. If nothing else, this thought experiment will prepare you to discuss tradeoffs within the system that tie directly back to the customer and organizational problems you are solving.
What’s this good for?
The first problem and solution descriptions plant the seed for the system’s design and provide something concrete for a feedback loop to operate on.
This first version is useful in spite of its incompleteness and errors. You can:
- come back to it a bit later and revise it easily
- show it to a colleague and perform an informal design review
- record this information in a design doc
- look at it while you start to create an electronic version of the diagram
- capture critical business and technical requirements
- visualize components, communication paths, data sinks and reason about “non-functional” requirements more effectively
- etc
Next, I’ll show how these initial steps in the design process lead to building out the design, gathering more feedback and testing the design.
#NoDrama
(1) Who said ‘simple as possible, but no simpler’ first? Einstein, Scroggins, Hirsch?
