In my last post I began building the business case for agile/iterative/incremental projects by examining the impact of complexity, specifically the convergence of three factors: Requirements, Technology and People. I described how these factors could be mapped on a modified Stacey Graph to illustrate the mismatch between the type of project and how it is managed and delivered.
This revealed that single-pass/phased waterfall delivery was best-suited to projects that had total agreement on requirements, total certainty on the technology used and team members who differed little in their personalities and approaches to problem-solving, ie. those whose constituent parts are well-understood. However, the reality for modern software projects is that there is significant disagreement on requirements, uncertainty in the technologies used and a team that has varied skills, abilities, personalities and other interpersonal issues.
They are inextricably complex and therefore unpredictable and will not respond well to the simple mechanisms of up-front planning and rigid marshalling through different phases of activity, no matter how much process, documentation and cracking the whip to “git ‘er done” is applied. The topic of this post is the examination of a solution to the problem of project complexity and unpredictability by adopting an alternative approach based on three “competitive power tools”: visibility, inspection and adaptation.
Defined vs. Empirical Process Control When we use a simple project management methodology like waterfall, we’re using what is known as a defined process control. We take this approach when we have little variance or complexity in the intermediate activities that go into building a product, like an assembly line: Insert tab A into slot B, bolt unit C on to chassis D, and so on.
All activities can be laid out and predicted with a reasonable degree of precision irrespective of the people involved and so are used when we want to mass produce products quickly and commoditize them (ie. give them a stable SKU or per unit price). However, as explained above, building custom software is not an easily commoditizable process because it is complex.
The intermediary processes involved are difficult if not impossible to lay out in a repeatable, predictable assembly-line where one follows a plan because the raw materials (user requirements) do not physically exist in a state that can be readily assembled into a final product.
They need to be deconstructed and translated into software through entirely intellectual effort. As a result, the end product is almost always a marginal-at-best representation of the thoughts of the development team and understanding of the technologies (eg. software, platforms, dependent external systems) involved. In light of these realities, it’s easy to see why software that is developed using a defined process control requires a significant amount of re-work and expense to get it into a finalized state.
In situations like this, we need to employ an alternative approach called empirical process control which emphasize three fundamental principles: Visibility, Inspection and Adaptation. Visibility:
- Within empirical process controlled projects, visibility is king; all aspects of the project that can affect outcome are completely transparent to the customer, stakeholders and team.
- Once a process is visible it can now be inspected with sufficient frequency to discover inconsistencies and problems that threaten the viability of the product and even the processes themselves.
- When we find a problem in our processes or product through inspection, empirical process controlled products allow for adaptations to be made to bring them into alignment with acceptable expectations and outcomes.
Scrum: Empirical Process Control in Action
Scrum is a direct implementation of empirical process control across all aspects of a software project. In contrast to the assembly line / defined process control approach of waterfall methodologies, the product and processes are planned, developed and reviewed in short, repeatable loops called Sprints that last 2-4 weeks.
The product and process are completely transparent thanks to daily team inspections (The Daily Scrum) and bi-weekly or monthly customer and team inspections (The Sprint Review and Sprint Retrospective).
The unpredictability and risks that are associated with software delivery are blunted or mitigated with Scrum because of the visibility of the process framework and its multiple inspection and adaptation points which do not exist in any meaningful form within a waterfall project. This gives customers and managers a competitive edge with timely access to information on the project state at any given time making adaptations possible when they can have the greatest impact. By making product inspection a regular, repeatable part of delivery with the Sprint Review, return-on-investment is promoted and protected: The product is either right or it is not and can be amended in successive Sprints to come into closer alignment with the expectations of the business. Given the alternatives, it’s clear why Scrum is a superior process framework and why I refer to it as a competitive power tool that can help accelerate time-to-market and improve product quality and thus dramatically boost ROI. This effect can be clearly seen when we compare and contrast the effect of multiple, inexpensive and shallow iterations over one deep and extremely expensive one:
However, process alone isn’t sufficient to make this possible: Working in an iterative/incremental model requires rethinking how we engage people in the software development effort and the roles they play. This will be the topic of my next post. What do you think? Let me know your thoughts below.