Planning in Phases
Intuitive, yet wrong for software development
Instead of trusting my intuition, sometimes I deliberately question it. I find that questioning a hunch can reveal a cognitive bias.
There’s one cognitive bias in particular that has gotten us software development professionals in a lot of trouble: taking an approach in one domain and applying it to another that appears analogous and similar.
The name for this could be functional fixedness, though I’m not entirely sure. I invite you to peruse the codex, if you’re so inclined.
Phased planning: an intuitive approach
How do we achieve ambitious goals that necessitate elaborate interactions between individuals and groups? How do we build complex things where many people are involved, and where many more things can go wrong?
We plan.
The instinct is to divide and conquer: to plan in phases. Except for the final one, each phase delivers an artifact that represents a precursor to the end result. Without the completion of the final phase, however, these artifacts are worthless. Consequently, the emphasis is on getting the planning right, so that everything falls into place in the final phase.
A home renovation example
Let me take you through an example of this approach. Recently, my wife and I had our bathroom renovated. The first phase — let’s call it the ideation phase — looked like this. We:
Sketched out how we want the room layout to be
Created a Pinterest mood board to articulate the desired aesthetics
Consulted with a designer, who helped narrow down our ideas
Subsequently, we moved to a budgeting phase where we:
Broke down all the trades and the work needed
Got quotes for said work and materials
Made minor adjustments to the design (output from ideation phase) so as to stay within budget
The third phase was the scheduling phase. Here, we:
Determined the sequence of work — what needed to be done first
Sourced and ordered off all the materials needed
Booked in the trades to do the work as appropriate to fit in timelines
The final implementation phase involved execution of the actual work, including on-the-spot adjustments to the plan. Here are some examples of what happened:
We had to get different light fixtures because the finish of the originals didn’t match
We discovered a pipe which required modifying the new vanity being installed
The contractor had to re-tile the bathtub wall due to a slanted ceiling, which resulting in a crooked appearance (fortunately we had extra tiles!)
While the final phase certainly yielded surprises, the adjustments to the plan were minor. In other words, we never felt that the overall goal was in jeopardy — we didn’t get the exact light fixtures we wanted, but we weren’t forced to abandon the entire project as a consequence.
I’m no expert in home renovations, but this approach seems pretty common for this field, and I imagine that it generally works. It’s also a pretty intuitive way to organize, plan, and execute projects in general. Projects such as events, vacations, and construction are often planned using a phased approach.
So then why doesn’t it work for software development?
Misapplication to software
Applying phased planning to software development could look something like this:
The stakeholder sets goals for the product in the form of success criteria. These could be:
A target market (e.g. SMB accountants)
A high-level product definition (e.g. cloud-based accounting software)
A budget (e.g. $1MM)
A timeline (e.g. one year)
The project manager breaks down the journey to this end-state in the form of phases. The phases might look something like:
Business Analysis — what the product will do
Architecture — how it will do it
Implementation — building the product
QA — fixing the product
The project manager devotes most of the project timeline to the planning phases — after all, planning mitigates the risk of surprises during the execution phases, which in turn can threaten the success of the project.
The planning phases are staffed with senior people — those with experience who know what can go wrong and who can anticipate and plan for it.
Consequently, most of the budget is consumed by the planning phases. Again, this is not a concern because, given that the plan is sufficiently well thought-out and defined, then execution should be nothing more than the completion of broken-down, well-defined, simple tasks.
Everyone assumes that no changes are needed to the product once the project is completed. Once the goals are achieved, the project has (by definition) succeeded and therefore, there will be nothing left to do.
Does this look familiar to you? I hope not, because this is the essence of waterfall software development. In particular, Big Design Up Front (BDUF) is a consequence of the long-running planning phases that waterfall incentivizes and ultimately entails.
Why phased planning fails for software development
Why can’t we have a phased approach with Agile software development? To quote the fourth value from the Agile manifesto:
Responding to change over following a plan
A phased approach is fragile to change, because it depends on outcomes from the prior phase. When conditions change, the prior phases need to be replayed. The artifacts from these phases must then be thrown out (as repurposing them always backfires).
In the home renovation example, we weren’t able to get the light fixture we wanted, but we were able to find a replacement that still works with the aesthetic we chose. You could say we were “Agile” there. Then again, it was a minor change. It’s easy to claim Agility when only faced with minor changes.
On the other hand, what if our plans encountered a major change? Imagine instead that we were rebuilding our patio, and the price of lumber skyrocketed just before we put the order in (and after all the trades had been scheduled). In this case, we’d have to scrap all of our carefully-laid plans, with nothing of actual value to show for all the planning work put in so far.
This is the degree of change that software development must respond to, and the degree of change that Agile software development can respond to.
Cyclical workflows: an alternative to phased planning
An Agile approach breaks down long-running, linear phases into short, cyclical workflows. Here are some examples of cyclical workflows commonly found in most Agile methodologies:
Iterations — weekly or bi-weekly
Team Standup — daily
Test-Driven Development (TDD) — several times a day
The point of each cycle is to deliver a unit of value, and then evaluate the attempt and result before the next cycle is played. For iterations, this unit of value is a product increment. For team standup, it’s a goal for the day. For TDD, it’s a working unit of code.
During software development, two fundamental questions arise: what to build, and how to build it. The phased approach tackles these questions by answering them in sequence. Agile tackles them by answering them continuously. It does so through the use of feedback loops1 enabled by cyclical workflows.
For example, “what to build” is answered through practices such as the Sprint Review in Scrum, build-measure-learn in Lean Startup, and on-site customer in XP. The common theme here is to inspect and adapt “what to build” as part of every cycle.
While not universally adopted, how to answer the question of “what to build” through an Agile lens is well-established. As stated, there are many excellent approaches out there, and many elegant elucidations of them.
Answering “how” continuously
The harder question is the second one: “how to build it?”
This is the riddle I will leave you with. How do we answer this question continually, vs. sequentially? How can we build part of something without knowing how the greater whole will be built?
I plan to address this conundrum in a subsequent post. In the meantime, I invite you to ponder over these questions, and to reflect on your software development workflow.
How do you tackle complex, technical challenges (e.g. challenges around software architecture)? Does your team work in phases? How much upfront planning is involved? Do you rely on your intuition to help you decide how and what to design, or have you adopted a well-reasoned process or practice?
Feedback loops in Agile are a more fundamental concept — and arguably a more important one — than cyclical workflows. However, I use the latter term here to emphasize the processes on Agile teams that are deliberately established to enable feedback loops. This is in contrast to the phases on waterfall teams that are deliberately established to attempt to plan for every possibility ahead of time.