Technical design process

When asked to make a change, most entry-level engineers jump straight into coding. Diving into code works at first, but you’ll eventually be given a task that’s too big to jump into; you’ll need to think about technical design.

Design work is broken into two activities: solitary deep-thought work and collaborative group discussion.

Researching, brainstorming, and writing make up the deep work.
Design discussions and commenting on design documents make up the collaborative part.

The tangible output of the process is a design document.

The Technical Design Process Cone

Designing software is not a linear process from research and brainstorming to documentation and approval. It is more like a spiral that alternates between independent and collaborative work, clarifying and refining the design at every step:

You begin at the base of the cone. You are unclear about the problem space, requirements, and possible solutions. So early in the process, it is impossible for you to have a solution that you’re confident in.
As you research, you bounce back and forth between independent work and discussion with a small group of teammates or experts in the domain you’re researching. You brainstorm and experiment. The goal is to learn—to increase certainty and clarity.
Eventually, your research, experimentation, and brainstorming leads you to a preferred design. After sanity-checking it with those you’ve been working with, you write a design document.
The arrow in the cone spirals further upward. You are now more certain that you understand the problem space. You have a design proposal and are ready to circulate it.
After your design is approved, implementation begins, but design isn’t over. Implementation will surface more surprises. You must update your design document if any large deviations are made as you and your team code.

Thinking About Design

Define the Problem:

Start by asking stakeholders what they perceive the problem to be.
Restate the problem, in your own words, back to the stakeholders. Ask if your understanding matches theirs. If there is more than one problem, ask which problems are the highest priority.
“What happens if we don’t solve this problem?” is a powerful question. When the stakeholder answers, ask if the outcome is acceptable. You’ll find many problems don’t actually need to be solved.
Once you’ve collected notes on the problem from various stakeholders, try to synthesize the feedback into a clear problem statement.

Do Your Research:

Don’t go straight from problem definition to “final” design.
Consider relevant research, alternative solutions, and trade-offs.
The design you come up with should be not your first idea but your best.

Conduct Experiments:

Experiment with your ideas by writing draft code and running tests. Write draft APIs and partial implementations. Run performance tests or even A/B user tests to learn how systems and users behave.
Don’t get attached to your experimental code. Proof-of-concept code is meant to illustrate an idea and be thrown away or rewritten.

Give It Time: Good design takes creativity. Don’t expect to sit down and bang out a design in one shot. Give yourself large chunks of time, take breaks, change your scenery, and be patient. A spike is an Extreme Programming (XP) term for a time-bounded investigation.

Writing Design Documents

Design documents are a scalable way to clearly communicate your ideas. The process of writing structures your thinking and highlights areas of weakness.

Document Consequential Changes:

Use these three criteria to decide if a design document is required:

The project will require at least a month of engineering work.
The change will have long-lasting implications with regard to extending and maintaining the software. Some changes are quick to introduce but come with long-term ramifications.
The change will significantly impact other teams. Changes that significantly impact many teams also need a design document. Teams need to know what you’re doing so they can provide feedback and so they can accommodate your changes.

Know Why You’re Writing: Superficially, design documents tell others how a software component works. But a design document’s function exceeds simple documentation. A design document is a tool to help you think, to get feedback, to keep your team informed, to ramp up new engineers, and to drive project planning.

Learn to Write: Writing is a skill, and like any other skill, it is developed through practice. Take advantage of opportunities to write—design documents, emails, code review comments—and try hard to write clearly.

Keep Design Documents Up-to-Date:

Two common pitfalls occur during the transition from proposal to documentation.

The first pitfall is that the proposal document is abandoned and never updated again. Implementations diverge, and the document is misleading to future users.
The second pitfall is that the document is updated, and the history of the proposal is lost.

Using a Design Document Template

A design document should describe the current code design, the motivation for a change, potential solutions, and a proposed solution. The document should include details about the proposed solution: architectural diagrams, important algorithmic details, public APIs, schemas, tradeoffs with alternatives, assumptions, and dependencies.

(Example:https://www.industrialempathy.com/posts/design-docs-at-google/)

Introduction: Introduce the problem being solved and say why it is worth solving. Provide a paragraph-long summary of the proposed change and some guidance that points different readers—security engineers, operations engineers, data scientists—to relevant sections.

Current State and Context: Describe the architecture that is being modified and define terminology. Talk about the ways in which the issue is currently being addressed. Are there workarounds being employed? What are their drawbacks?

Motivation for Change: Software teams tend to have more projects than they can tackle at once. Why is this particular problem worth solving, and why now? But be careful not to overpromise!

Requirements: List requirements that an acceptable solution must meet. These can be broken out into sections such as the following:

User-facing requirements: These are usually the bulk of the requirements. They define the nature of the change from a user perspective.
Technical requirements: These include hard requirements on the solution that must be met. Technical requirements are usually caused by interoperability concerns or strict internal guidelines, such as “Must support MySQL for the storage layer” or “Must provide an OpenAPI spec to work with our application gateway.” Service level objectives can also be defined here.
Security and compliance requirements
Other: This can include critical deadlines, budgets, and other important considerations.

Potential Solutions: There are usually multiple ways one can solve a problem. Writing this section is as much a tool for you as the reader; it’s meant to force you to think through not just your first idea but alternative ideas and their trade-offs. Describe reasonable alternative approaches and why you dismissed them. Describing potential solutions will preemptively address “Why not do X?” comments.

Proposed Solution: Describe the solution you settled on. This description goes into more detail than the very brief description in the introduction and may contain diagrams that highlight changes. Here and in the following sections, if your proposal includes multiple phases, explain how the solution evolves from phase to phase.

Design and Architecture: Design and architecture normally make up the bulk of the document. All the technical minutiae worth discussing go here.

System Diagram: Include a diagram that shows the main components and how they interact.
UI/UX Changes: Create mock-ups if your project changes user interfaces. Use the mocks to walk through a user’s activity flow. If your change does not have a visual component, this section might talk about developer experience with the library you are creating or ways a user might use your command line tool.
Code Changes: Describe your implementation plan. Highlight what, how, and when existing code will need to change. Describe any new abstractions that need to be introduced.
API Changes: Document changes to existing APIs and any newly proposed APIs. Discuss backward/forward compatibility and versioning. Remember to include error handling in your API proposal: it should respond with useful information when encountering malformed inputs, constraint violations, and unexpected internal errors or exceptions.
Persistence Layer Changes.

Test Plan: Do not define every test in advance; rather, explain how you plan to verify your changes.

Rollout Plan: Describe the strategies you’ll use to avoid complicated deployment ordering requirements. Document the feature flags you will need to put in place to control the rollout. Think about how you would find out if changes are not working and how you will roll back if problems are discovered.

Unresolved Questions: This is a good way to solicit input from your readers and to state your “known unknowns.”

Appendix: This is also a good place to add references to related work and further reading.

Collaborating on Design

Understand Your Team’s Design Review Process:

Architectural reviews are more formal, heavier-weight processes. Designs must be approved by outside stakeholders such as operations and security members. A design document is required, and there might be multiple rounds of meetings or presentations. Because of their high time cost, architectural reviews are reserved for large or risky changes. Don’t wait on final approval before writing code. Spend time implementing prototypes and proof-of-concept “spikes” to increase confidence in the design and to give you a shorter path to production. But don’t go beyond proof-of-concept work.
We call the other type of design review process request for decision, or RFD (not to be confused with the Internet Society request for comment process, the RFC). The term RFD is not very common, but the pattern is; RFDs are fast intrateam reviews to quickly reach decisions that need some discussion but not a full review.

Don’t Surprise People: Gently and incrementally ease people into your design proposal. You’re setting yourself up for failure if a formal design document is the first time other teams and tech leads learn of your work. Each party has a different point of view and a different set of interests and may react strongly to a sudden design document that they had no say in. Feedback sessions don’t need to be formal or scheduled. Casual conversations over lunch, in a hallway, or before meetings start are fine—even preferred.

Brainstorm with Design Discussions: Design discussions help you understand a problem space, share knowledge, discuss trade-offs, and solidify design. These brainstorming sessions are informal, conversation is free flowing, and whiteboards are filled with ink. Brainstorming sessions range in size from two to about five.

Contribute to Design: You should contribute to your team’s design work, not just your own. As with code reviews, contributing to design can feel uncomfortable. You might think you have nothing to contribute to designs from more senior developers. Reading design documents and going to brainstorming meetings might feel like a distraction. Do it anyway. Your participation will improve your team’s design and help you learn.

Note

Sources: